Cpp-Bitcoin v22.0: Bitcoin Core integration/staging tree

icon
Latest Release: v22.0

Bitcoin Core version 22.0 is now available from:

https://bitcoincore.org/bin/bitcoin-core-22.0/

For the release notes please see the git repository:

https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-22.0.md

Do not use the links provided by GitHub, rather use the above download links, they are guaranteed to be generated deterministically and signed.

Source code(tar.gz)
Source code(zip)

Bitcoin Core integration/staging tree

Build Status

https://bitcoincore.org

What is Bitcoin?

Bitcoin is an experimental digital currency that enables instant payments to anyone, anywhere in the world. Bitcoin uses peer-to-peer technology to operate with no central authority: managing transactions and issuing money are carried out collectively by the network. Bitcoin Core is the name of open source software which enables the use of this currency.

For more information, as well as an immediately useable, binary version of the Bitcoin Core software, see https://bitcoin.org/en/download, or read the original whitepaper.

License

Bitcoin Core is released under the terms of the MIT license. See COPYING for more information or see https://opensource.org/licenses/MIT.

Development Process

The master branch is regularly built and tested, but is not guaranteed to be completely stable. Tags are created regularly to indicate new official, stable release versions of Bitcoin Core.

The contribution workflow is described in CONTRIBUTING.md.

The developer mailing list should be used to discuss complicated or controversial changes before working on a patch set.

Developer IRC can be found on Freenode at #bitcoin-core-dev.

Testing

Testing and code review is the bottleneck for development; we get more pull requests than we can review and test on short notice. Please be patient and help out by testing other people's pull requests, and remember this is a security-critical project where any mistake might cost people lots of money.

Automated Testing

Developers are strongly encouraged to write unit tests for new code, and to submit new unit tests for old code. Unit tests can be compiled and run (assuming they weren't disabled in configure) with: make check. Further details on running and extending unit tests can be found in /src/test/README.md.

There are also regression and integration tests of the RPC interface, written in Python, that are run automatically on the build server. These tests can be run (if the test dependencies are installed) with: qa/pull-tester/rpc-tests.py

The Travis CI system makes sure that every pull request is built for Windows, Linux, and OS X, and that unit/sanity tests are run automatically.

Manual Quality Assurance (QA) Testing

Changes should be tested by somebody other than the developer who wrote the code. This is especially important for large or high-risk changes. It is useful to add a test plan to the pull request description if testing the changes is not straightforward.

Translations

Changes to translations as well as new translations can be submitted to Bitcoin Core's Transifex page.

Translations are periodically pulled from Transifex and merged into the git repository. See the translation process for details on how this works.

Important: We do not accept translation changes as GitHub pull requests because the next pull from Transifex would automatically overwrite them again.

Translators should also subscribe to the mailing list.

Comments

  • [POC] build: enable Pointer Authentication and Branch Target Identification for aarch64 (Linux)
    [POC] build: enable Pointer Authentication and Branch Target Identification for aarch64 (Linux)

    Jan 21, 2022

    Arm Pointer Authentication (PAC) is a method of hardening code from Return Oriented Programming (ROP) attacks. It uses a tag in a pointer to sign and verify pointers. Branch Target Identification (BTI) is another code hardening method, where the branch/jump target is identified with a special landing pad instruction. Outside of some system support in glibc+kernel, packages gain the additional hardening by compiling with the -mbranch-protection=flag available in recent versions of GCC. In particular -mbranch-protection=standard enables both BTI and PAC, with backwards compatible to armv8.0 code sequences that activate on v8.3 (PAC) & v8.5 (BTI) enabled Arm machines. (taken from Fedora).

    Requirements for building/running with these features:

    • Compiler support for -mbranch-protection= flag:
      • Introduced in GCC 9.1 (our minimum required is 8). GCC 8 has a (now deprecated) msign-return-address flag.
      • Introduced in Clang 8 (our minimum required is 7).
    • (maybe) Linker support for -force-bti && -pac-plt:
      • binutils ld has supported both since it's 2.33.1 release.
      • LLVM 9 lld had a --pac-plt, which became -z,pac-plt in LLVM 10. More info.
    • TODO: Kernel / runtime requirements?

    Creation of a BTI enabled binary also requires that everything being linked in be BTI enabled. This means you currently cannot, for example, cross-compile using a Ubuntu based aarch64 toolchain, if you're wanting to use this feature. This can be shown using -Wl,z,force-bti, which will emit warnings for linked objects that are not BTI enabled (this is used in configure to detect when to disable using the flags). i.e:

    int main() { return 0; }
    
    # gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
    aarch64-linux-gnu-g++ test.cpp -mbranch-protection=standard -Wl,-z,force-bti
    /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/lib/../lib/Scrt1.o: warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
    /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/lib/../lib/crti.o: warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
    /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /usr/lib/gcc-cross/aarch64-linux-gnu/9/crtbeginS.o: warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
    /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /usr/aarch64-linux-gnu/lib/libc_nonshared.a(elf-init.oS): warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
    /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /usr/lib/gcc-cross/aarch64-linux-gnu/9/crtendS.o: warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
    /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/bin/ld: /usr/lib/gcc-cross/aarch64-linux-gnu/9/../../../../aarch64-linux-gnu/lib/../lib/crtn.o: warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
    

    However, if you compile on a system where the toolchain has been built with the additional hardening, i.e Fedora 33 and onwards:

    # gcc version 11.2.1 20210728 (Red Hat 11.2.1-1) (GCC)
    aarch64-redhat-linux-g++ test.cpp -mbranch-protection=standard -Wl,-z,force-bti
    ...
    # readelf -n a.out 
    
    Displaying notes found in: .note.gnu.property
      Owner                Data size 	Description
      GNU                  0x00000010	NT_GNU_PROPERTY_TYPE_0
          Properties: AArch64 feature: BTI, PAC
    

    Note the BTI and PAC properties. More about Fedora use of -mbranch-protection=standard by throughout it's packages can be seen in the rpc repos. i.e: Part of the base compiler flags for aarch64 && Used by default when building glibc.

    I've built and tested binaries on an aarch64 machine, (Neoverse-N1, Armv8.2-A) running Fedora 34.

    make -C depends/ -j5 NO_QT=1 NO_WALLET=1 NO_ZMQ=1 NO_UPNP=1 NO_NATPMP=1
    ./autogen.sh
    CONFIG_SITE=/home/fedora/bitcoin/depends/aarch64-unknown-linux-gnu/share/config.site ./configure
    make -j9
    ...
    /usr/bin/ld: secp256k1/.libs/libsecp256k1.a(libsecp256k1_la-secp256k1.o): warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
    /usr/bin/ld: secp256k1/.libs/libsecp256k1.a(libsecp256k1_la-secp256k1.o): warning: BTI turned on by -z force-bti when all inputs do not have BTI in NOTE section.
    

    Unit and functional tests pass. Note section contains (not no PAC):

    readelf -n src/bitcoind
    ...
      Owner                Data size 	Description
      GNU                  0x00000010	NT_GNU_PROPERTY_TYPE_0
          Properties: AArch64 feature: BTI
    

    I am running a sync with -assumevalid=0. However given that this machine does not support PAC or BTI, as it's Armv8.2, I'll have to find some newer aarch64 hardware to test other things on. Although this is still a demonstration that the PAC / BTI instructions are nops when running on older hardware.

    Further reading: https://fedoraproject.org/wiki/Changes/Aarch64_PointerAuthentication https://developer.arm.com/documentation/102433/latest/Return-oriented-programming https://lwn.net/Articles/789370/

    Related to #19075. Still a WIP. Haven't looked at anything similar for arm64 (M1).

    Linux/Unix Build system 
    Reply
  • p2p: Replace RecursiveMutex `cs_tx_inventory` with Mutex and rename it
    p2p: Replace RecursiveMutex `cs_tx_inventory` with Mutex and rename it

    Jan 21, 2022

    This PR is related to #19303 and gets rid of the RecursiveMutex cs_tx_inventory and also adds AssertLockNotHeld macros combined with LOCKS_EXCLUDED thread safety annotations to avoid recursive locking.

    P2P 
    Reply
  • wallet: BIP 326 sequence based anti-fee-snipe for taproot inputs
    wallet: BIP 326 sequence based anti-fee-snipe for taproot inputs

    Jan 22, 2022

    The goal of BIP 326 is to provide a "privacy cloak" for txs that use sequence-based locking. This is only possible for taproot inputs, as taproot spends may look identical for "dumb wallets" and "smart contracts" iff they use the key-path spend.

    Sequence-based locking has minimally lower anti-fee sniping guarantees if all the txs that created the inputs aren't itself locked to the block they were included in. However, the minimal degradation should be acceptable, given that anti-fee-snipe will set the locktime to the past of 10% of the txs anyway.

    Wallet RPC/REST/ZMQ 
    Reply
  • build: Fix xargs warnings for Guix builds
    build: Fix xargs warnings for Guix builds

    Jan 22, 2022

    On master (e3ce019667fba2ec50a59814a26566fb67fa9125) there are warnings in ./contrib/guix/guix-build logs:

    xargs: warning: options --max-args and --replace/-I/-i are mutually exclusive, ignoring previous --max-args value
    

    This PR fixes such warnings.

    Build system Scripts and tools 
    Reply
  • doc: Update the used Qt version
    doc: Update the used Qt version

    Jan 23, 2022

    Missed in bitcoin/bitcoin#23489.

    Docs 
    Reply
  • build, qt: Fix Windows cross-compiling with Qt 5.15
    build, qt: Fix Windows cross-compiling with Qt 5.15

    Jan 23, 2022

    While changes introduced in bitcoin/bitcoin#22093 worked fine with Qt 5.12, after bumping Qt up to 5.15 the cross-compiling of qt package for Windows fails with error: ‘mutex’ in namespace ‘std’ does not name a type.

    The first commit fixes this bug.

    The second commit cleans up a related CI script.

    The third commit improves related docs (see https://github.com/bitcoin/bitcoin/pull/22093#discussion_r680911586).

    Build system 
    Reply
  • build: Bump minimum Qt version to 5.11.3
    build: Bump minimum Qt version to 5.11.3

    Jan 23, 2022

    The current minimum Qt version is 5.9.5 which has been set in bitcoin/bitcoin#21286.

    Distro support:

    As another Ubuntu LTS is coming soon, it seems unreasonable to stick to Qt 5.9 which support ended on 2020-05-31. Anyway, it's still possible to build Bitcoin Core GUI with depends on bionic system.

    Bumping the minimum Qt version allows to make code safer and more reliable, e.g.:

    This PR is intended to be merged early after branching 23.x off.

    GUI Build system 
    Reply
  • index: Improve robustness of coinstatsindex at restart
    index: Improve robustness of coinstatsindex at restart

    Jan 23, 2022

    This change lets the coinstatsindex fail loudly in case the internal muhash state differs from the last finalized output saved on disk, which would indicate that the muhash state somehow got out of sync. This should generally not happen since both are written to disk in a batch but #24076 seems to indicate that the might still be an issue.

    Since #24076 so far can not be reproduced reliably, the issue should not be closed yet. Further investigation and testing needs to be done.

    UTXO Db and Indexes 
    Reply
  • build: Fix zeromq package when cross-compiling
    build: Fix zeromq package when cross-compiling

    Jan 23, 2022

    Since bitcoin/bitcoin#23956 on some systems (impish, jammy) zeromq package build system erroneously detects libbsd package from the host system:

    --- a/libzmq.pc
    +++ b/libzmq.pc
    @@ -8,5 +8,5 @@
     Version: 4.3.4
     Libs: -L${libdir} -lzmq
     Libs.private:  -lpthread
    -Requires.private: 
    +Requires.private:  libbsd
     Cflags: -I${includedir} 
    

    This causes the configure fails to detect the zeromq package:

    configure: WARNING: libzmq version 4.x or greater not found, disabling
    

    This PR fixes this issue.

    Please note that more general solution was suggested in 9a3194cd666b390bc4c520f8291ab2b673061cae (bitcoin/bitcoin#19882).

    Build system 
    Reply
  • Extract CTxIn::MAX_SEQUENCE_NONFINAL constant, rework BIP 65/68/112 docs
    Extract CTxIn::MAX_SEQUENCE_NONFINAL constant, rework BIP 65/68/112 docs

    Jan 24, 2022

    Extracting the constant makes it possible to attach documentation to it.

    Also, rework the docs for the other "sequence constants".

    Refactoring Docs 
    Reply
  • LevelDB corruption with 0.8.x Mac client
    LevelDB corruption with 0.8.x Mac client

    Jun 14, 2013

    I've run bitcoin-qt Mac client on the latest OSX version since 0.3.x thru to 0.7.2 with literally zero problems ever, on essentially this same hardware setup.

    Since the 0.8 update and switch to LevelDB none of the three Mac client releases have worked stably for me and I've had to downgrade to 0.7.2 (with the May 15 workaround) to maintain a stable bitcoin wallet.

    Current setup is: MacBook Pro Retina, OSX 10.8.4, bitcoin-qt 0.8.2

    I saw that some of the known corruption issues/bugs were fixed/closed with the 0.8.2 release, so I decided to try the upgrade again. After re-indexing and working fine for hours at time (much better than 0.8.1 at least) upon restart I get "Error opening block database. Do you want to rebuild the block database?" which of course I don't want to do because it takes forever, even on this world's fastest Mac with SSD drive. This has happened 6 times now, and the interesting line in debug.log is:

    Verifying last 288 blocks at level 3 LevelDB read failure: Corruption: block checksum mismatch

    More details here: https://bitcointalk.org/index.php?topic=155140.0

    Downgrading to 0.7.2 again, please help permanently fix this in the next 0.8.x release.

    Bug 
    Reply
  • BIP-325: Signet [consensus]
    BIP-325: Signet [consensus]

    Mar 5, 2020

    This PR is a part of BIP-325 (https://github.com/bitcoin/bips/blob/master/bip-0325.mediawiki), and is a sub-PR of #16411.

    • Signet consensus (this)
    • Signet RPC tools (pending)
    • Signet utility scripts (contrib/signet) (pending)
    Validation Consensus 
    Reply
  • Add normalized transaction hash
    Add normalized transaction hash

    Feb 11, 2014

    This is just a very basic implementation for exposure/review, with lacking documentation, and no functionality to look up transactions by normalized txid.

    Reply
  • Relay and alert user to double spends
    Relay and alert user to double spends

    Mar 16, 2014

    Rebased version of #3354.

    Instead of dropping double-spend transactions, relay them to peers. This is crucial to allowing all parts of the network to detect double-spend attempts as quickly as possible, i.e. reducing the "confirmation" time for buying coffee to a time on par with credit card transactions. Only the first respend seen is relayed.

    I successfully completed a primary test running 3 copies of bitcoin-qt in regtest mode with this patch applied, connected in a node1-newtorknode-node2 configuration.

    I used the rawtransaction API for steps 2 and 3, but with -salvagewallet and a single funding transaction, tests would be possible that did not require rawtransactions.

    Still TODO: Regression test plan

    Primary Test Details Step 1

    • Send transaction from node1 to a node2 address
    • * Transaction is relayed through networknode, to node2
    • * Transaction appears in node2 wallet

    Step 2

    • Restart node1 with -salvagewallet (so it forgets about 1st spend)
    • Send new transaction using same input (first double-spend), using rawtransaction API
    • * Transaction is relayed through networknode, to node2
    • * node2 does not accept the double-spend into its mempool and it does not appear in node2 wallet

    Step 3

    • Restart node1 with -salvagewallet again
    • Send transaction using same input (second double spend), using rawtransaction API
    • * Transaction is not relayed through networknode

    Step 4 One more test for good measure

    • Restart node1 with -salvagewallet again
    • Using regular UI (not rawtransaction API), send entire wallet balance to node2 (third double spend). This included additional inputs besides the spent one.
    • * Transaction is not relayed through networknode
    Reply
  • Yet another Coin Control Release
    Yet another Coin Control Release

    Mar 4, 2013

    https://bitcointalk.org/index.php?topic=144331

    Reply
  • ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large
    ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large

    Jan 15, 2015

    Client info

    2015-01-15 18:15:21 Bitcoin version v0.10.99.0-4f73a8f-dirty (2015-01-09 20:21:19 -0800) 2015-01-15 18:15:21 Using OpenSSL version OpenSSL 1.0.1f 6 Jan 2014 2015-01-15 18:15:21 Using BerkeleyDB version Berkeley DB 5.3.28: (September 9, 2013) 2015-01-15 18:15:21 Default data directory /home/fredrik/.bitcoin 2015-01-15 18:15:21 Using data directory /home/fredrik/.bitcoin 2015-01-15 18:15:21 Using config file /home/fredrik/.bitcoin/bitcoin.conf 2015-01-15 18:15:21 Using at most 200 connections (1024 file descriptors available) 2015-01-15 18:15:21 Using 0 threads for script verification 2015-01-15 18:15:21 Binding RPC on address ::1 port 8332 (IPv4+IPv6 bind any: 0) 2015-01-15 18:15:21 Binding RPC on address 127.0.0.1 port 8332 (IPv4+IPv6 bind any: 0) 2015-01-15 18:15:21 Using wallet wallet.dat 2015-01-15 18:15:21 init message: Verifying wallet... 2015-01-15 18:15:21 CDBEnv::Open : LogDir=/home/fredrik/.bitcoin/database ErrorFile=/home/fredrik/.bitcoin/db.log 2015-01-15 18:15:21 Bound to [::]:8333 2015-01-15 18:15:21 Bound to 0.0.0.0:8333 2015-01-15 18:15:21 init message: Loading block index... 2015-01-15 18:15:21 Opening LevelDB in /home/fredrik/.bitcoin/blocks/index 2015-01-15 18:15:22 Opened LevelDB successfully 2015-01-15 18:15:22 Opening LevelDB in /home/fredrik/.bitcoin/chainstate 2015-01-15 18:15:28 Opened LevelDB successfully 2015-01-15 18:15:32 LoadBlockIndexDB: last block file = 218 2015-01-15 18:15:32 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=274, size=115501816, heights=338812...339085, time=2015-01-13...2015-01-15) 2015-01-15 18:15:32 Checking all blk files are present... 2015-01-15 18:15:32 LoadBlockIndexDB(): transaction index disabled 2015-01-15 18:15:32 LoadBlockIndexDB(): hashBestChain=0000000000000000188de542fd76b1676c4be6c380b39ddea119358c290cebd7 height=339085 date=2015-01-15 18:07:14 progress=0.999987 2015-01-15 18:15:32 init message: Verifying blocks... 2015-01-15 18:15:32 Verifying last 288 blocks at level 3

    log before crash

    2015-01-13 22:15:27 UpdateTip: new best=0000000000000000159e7d66c954312bccb5f12de808f2991b3859205665c442 height=338819 log2_work=81.997891 tx=56658614 date=2015-01-13 22:14:41 progress=0.999999 cache=6969 2015-01-13 22:15:58 ResendWalletTransactions() 2015-01-13 22:17:06 receive version message: /Satoshi:0.9.2/opennodes.org:0.1/: version 70002, blocks=338819, us=84.215.171.162:8333, peer=2855 2015-01-13 22:17:25 receive version message: /libbitcoin:2.0/: version 60000, blocks=0, us=10.0.0.1:8333, peer=2856 2015-01-13 22:17:25 Added time data, samples 200, offset -5 (+0 minutes) 2015-01-13 22:18:06 socket recv error Connection reset by peer (104) 2015-01-13 22:18:13 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=2857 2015-01-13 22:18:31 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=338818, us=84.215.171.162:8333, peer=2858 2015-01-13 22:19:16 receive version message: /Satoshi:0.9.3/: version 70002, blocks=338817, us=84.215.171.162:8333, peer=2859 2015-01-13 22:19:16 Added time data, samples 200, offset -2 (+0 minutes) 2015-01-13 22:19:32 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=2860 2015-01-13 22:19:47 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=2861 2015-01-13 22:22:36 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=338819, us=84.215.171.162:8333, peer=2862 2015-01-13 22:22:44 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:22:45 receive version message: /BTCXchange.ro Mapper:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=2863 2015-01-13 22:22:47 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=338819, us=127.0.0.1:8333, peer=2864 2015-01-13 22:22:47 Added time data, samples 200, offset +10 (+0 minutes) 2015-01-13 22:23:00 receive version message: /Satoshi:0.8.4/: version 70001, blocks=338819, us=84.215.171.162:8333, peer=2865 2015-01-13 22:23:00 Added time data, samples 200, offset -7 (+0 minutes) 2015-01-13 22:23:53 receive version message: /Satoshi:0.8.2.2/: version 70001, blocks=336635, us=84.215.171.162:8333, peer=2866 2015-01-13 22:23:53 Added time data, samples 200, offset +14 (+0 minutes) 2015-01-13 22:24:13 receive version message: /Satoshi:0.9.3/: version 70002, blocks=338735, us=84.215.171.162:8333, peer=2867 2015-01-13 22:24:13 Added time data, samples 200, offset -5 (+0 minutes) 2015-01-13 22:25:06 receive version message: /Snoopy:0.1/: version 60001, blocks=0, us=84.215.171.162:8333, peer=2868 2015-01-13 22:25:49 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=2869 2015-01-13 22:26:33 UpdateTip: new best=00000000000000001829e20898eaff72aa667d3715a94535afae5ae7ada07bc0 height=338820 log2_work=81.997947 tx=56659414 date=2015-01-13 22:25:41 progress=0.999999 cache=9596 2015-01-13 22:26:35 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:26:43 ERROR: AcceptToMemoryPool : nonstandard transaction: dust 2015-01-13 22:26:48 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:26:58 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:27:01 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=84.215.171.162:8333, peer=2870 2015-01-13 22:27:20 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:27:22 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:27:46 UpdateTip: new best=000000000000000008b332686e0043a4b95ca7e46d1b0785536981b543dfc755 height=338821 log2_work=81.998004 tx=56659415 date=2015-01-13 22:36:09 progress=1.000013 cache=9706 2015-01-13 22:28:29 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:28:34 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:28:48 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:28:52 receive version message: /getaddr.bitnodes.io:0.1/: version 70002, blocks=338819, us=84.215.171.162:8333, peer=2871 2015-01-13 22:29:22 ERROR: AcceptToMemoryPool : inputs already spent 2015-01-13 22:30:29 ERROR: ReadBlockFromDisk : Deserialize or I/O error - ReadCompactSize() : size too large 2015-01-15 18:02:07

    UTXO Db and Indexes Data corruption 
    Reply
  • Treat dust outputs as non-standard, un-hardcode TX_FEE constants
    Treat dust outputs as non-standard, un-hardcode TX_FEE constants

    Apr 26, 2013

    This is a quick, safe fix for transaction fee handling. A riskier, more complicated fix still needs to happen, but will take more time to code/test/etc.

    Two motivations for this pull:

    First, to discourage people from bloating users' wallets and the UTXO set with tiny outputs.

    This pull defines 'uneconomic dust' as 54.3 uBTC (5430 satoshis, about $0.007 at current prices), and treats any transaction with outputs less than 5430 satoshis as non-standard (won't be relayed, won't be mined). 5430 satoshis is derived from the cost (in fees) to spend a TxOut/TxIn. See https://people.xiph.org/~greg/txouts2.png for proportion of recent outputs this will (eventually) affect.

    Second, we have no idea what will happen to Bitcoin prices or transaction volume / competition for space in blocks. So this patch lets users and miners specify the minimum transaction fees at startup, using the -mintxfee / -mintxrelayfee options (which I'm intentionally leaving undocumented for now). The dust/nonstandard test is based on the -mintxrelayfee.

    Qt and RPC both now tell the user why CreateTransaction fails, if it fails; Qt error reporting is a little wonky (try to send one satoshi, and you get two modal dialog boxes, one after the other; I don't care enough to try to fix that).

    Note: I spent several hours trying, and failing, to create unit tests for this patch; CWallet::fFileBacked is ignored by several of the wallet routines used by CWallet::CreateNewTransaction. In the end, I decided thatrefactoring CWallet to support unit testing would be much more extensive and riskier than these changes.

    Reply
  • Add ZeroMQ notifications
    Add ZeroMQ notifications

    May 4, 2015

    Continues Johnathan Corgan and Jeff Garzik's work. Supercedes #4594 and #5303. As discussed in #6072 and #5328.

    Feature 
    Reply
  • Add a getutxos command to the p2p protocol
    Add a getutxos command to the p2p protocol

    Jun 16, 2014

    Introduction

    The getutxo command allows querying of the UTXO set given a set of of outpoints. It has a simple implementation and the results are not authenticated in any way. Despite this, there are times when it is a useful capability to have. I believe @jgarzik also has a use case for this, though I don't know what it is.

    As a motivating example I present Lighthouse, an app I'm writing that implements assurance contracts:

    http://blog.vinumeris.com/2014/05/17/lighthouse/

    Lighthouse works by collecting pledges, which contain an invalid transaction signed with SIGHASH_ANYONECANPAY. Once sufficient pledges are collected to make the combination valid, we say the contract is complete and it can be broadcast onto the network, spending the pledged outputs. Before that occurs however, a pledge can be revoked and the pledged money redeemed by double spending the pledged output. For instance you might want to do this if it becomes clear not enough people care about the assurance contract for it to reach its goal in a timely manner, or if you simply need the money back due to some kind of cashflow crunch.

    It is convenient to be able to see when a pledge has been revoked, so the user interface can be updated, and so when the final contract is created revoked pledges can be left out. For this purpose "getutxos" is used.

    Protocol

    The getutxos message takes a boolean which controls whether outputs in the memory pool are considered, and a vector of COutPoint structures. It returns a bitmap with the same number of bits as outputs specified rounded up to the nearest 8 bits, and then a list of CTxOut structures, one for each set bit in the bitmap. The bitmap encodes whether the UTXO was found (i.e. is indeed unspent).

    Authentication

    The results of getutxos is not authenticated. This is because the obvious way to do this requires the work maaku has been doing on UTXO commitments to be merged, activated by default, miners to upgrade and a forking change made to enforce their accuracy. All this is a big project that may or may not ever come to fruition.

    For the Lighthouse security model however, this doesn't matter much. The reason is that the pledge transactions you're getting (which may be malicious) don't come from the P2P network. They come in the form of files either from a simple rendezvous server, or e.g. a shared folder or email attachments. The people sending these files have no way to influence the choice of peers made by the app. Once the outputs are returned, they are used to check the signatures on the pledge, thus verifying that the pledge spends the UTXO returned by the P2P network.

    So we can be attacked in the following ways:

    • The pledge may be attempting to pledge non-existent outputs, but this will be detected if the majority of peers are honest.
    • The peers may be malicious and return a wrong or bogus output, but this will be detected when the signature is checked, except for the value (!) but we want to fix this by including the value into the sighash at some point anyway because we need it to make the TREZOR efficient/faster.
    • The peers may bogusly claim no such UTXO exists when it does, but this would result in the pledge being seen as invalid. When the project creator asks the pledgor why they revoked their money, and learns that in fact they never did, the bogus peers will be detected. No money is at risk as the pledges cannot be spent.
    • If the pledgor AND all the peers collaborate (i.e. the pledgor controls your internet connection) then they can make you believe you have a valid pledge when you don't. This would result in the app getting "jammed" and attempting to close an uncloseable contract. No money is at risk and the user will eventually wonder why their contract is not confirming. Once they get to a working internet connection the subterfuge will be discovered.

    There is a final issue: the answer to getutxos can of course change the instant the result is generated, thus leading you to construct an uncloseable transaction if the process of revocation races with the construction. The app can detect this by watching for either a reject message, or an inv requesting one of the inputs that is supposed to be in the UTXO set (i.e. the remote peer thinks it's an orphan). This can then cause the app to re-request the UTXO set and drop the raced pledge.

    In practice I do not anticipate such attacks are likely to occur, as they're complicated to pull off and it's not obvious what the gain is.

    There may be other apps that wish to use getutxos, with different security models. They may find this useful despite the lack of UTXO commitments, and the fact that the answer can change a moment later, if:

    • They are connecting to a trusted peer, i.e. localhost.
    • They trust their internet connection and peer selection, i.e. because they don't believe their datacenter or ISP will commit financial fraud against them, or they are tunnelling via endpoints they trust like a randomly chosen Tor exit.
    • They are not using the response for anything important or worth attacking, like some kind of visualisation.

    Upgrade

    If enforced UTXO commitments are added to the block chain in future, it would make sense to rev the P2P protocol to add the proofs (merkle branches) to the response.

    Testing

    I attempted to write unit tests for this, but Core has no infrastructure for building test chains .... the miner_tests.cpp code does it but at the cost of not allowing any other unit test to do so, as it doesn't reset or clean up the global state afterwards! I tried to fix this and ended up down a giant rabbit hole.

    So instead I've tested it with a local test app I wrote, which also exercises the client side part in bitcoinj.

    BIP

    If the code is ACKd then I will write a short BIP explaining the new message.

    Philosophy

    On IRC I have discussed this patch a little bit before. One objection that was raised is that we shouldn't add anything to the P2P protocol unless it's unattackable, because otherwise it's a sharp knife that people might use to cut themselves.

    I personally disagree with this notion for the following reasons.

    Firstly, many parts of the P2P protocol are not completely unattackable: malicious remote nodes can withhold broadcast transactions, invent fictional ones (you'd think they're orphans), miss out Bloom filter responses, send addr messages for IP's that were never announced, etc. We shouldn't hold new messages to a standard existing messages don't meet.

    Secondly, even with UTXO commitments in the block chain, given the sick state of mining this only requires a collaboration of two people to undo, although that failure would be publicly detectable which is at least something. But anyway, there's a clean upgrade path if/when UTXO authentication becomes available.

    Thirdly, we have a valid use case that's actually implemented. This isn't some academic pie in the sky project.

    Finally, Bitcoin is already the sharpest knife imaginable. I don't think we should start rejecting useful features on the grounds that someone else might screw up with them.

    If the above analysis ends up not holding for some reason, and people do get attacked due to the lack of authentication, then Lighthouse and other apps can always fall back to connecting to trusted nodes (perhaps over SSL). But I would like to optimistically assume success up front and see what happens, than pessimistically assume the worst and centralise things up front.

    P2P 
    Reply
  • BIP-68: Mempool-only sequence number constraint verification
    BIP-68: Mempool-only sequence number constraint verification

    Jun 19, 2015

    Essentially enforces sequence numbers as a relative lock-time as an IsStandard() rule to make it easy to test applications using it; blocks containing transactions with invalid sequence numbers given the age of their inputs per BIP 68 are still accepted.

    This pull-req is not a soft-fork nor does it contain any code to start that process.

    This pull request builds on top of #6177.

    Background: BIP 68, Consensus-enforced transaction replacement signalled via sequence numbers

    Feature Mempool 
    Reply