Second development efforts

Search…
This section describes the efforts and work done by the Yale Openlab team from September 2019 to February 2020
The Data (Consensus) Challenge on blockchains
Building an accounting system on blockchain for external assets is exactly as difficult as it sounds. One of our first tasks was to populate data in to our DB. The only problem with this was the datasets were completely incompatible. We want to develop a nested accounting structure where everybody can verify datasets and algorithms. To meet this challenge we started writing a postgres database and blockchain state to manage it. 
Public blockchains are meant to be decentralized and the way they establish consensus around information is agreed upon in the consensus engine that block producers must run. One of the stranger concepts to deal with here is integrity of the datasets. In order to provide our climate data that data has to be on-boarded on to the platform. Which means the identity of the transaction signer should be known and the sources referenced. This paradigm allows the whole community to verify identity and provide a point of view based approach to data confidence.
Traditionally you may publish your work in a journal, this process allows a social graph to verify your identity, your academic and professional relationships, the methods you've used and finally the data you've produced to help others understand your work. In this way other can reproduce your work and provide consensus via reproducible results. 
The first hurdle here is an identity graph. Our approach will be to let anybody register a keypair with any name and email. Once a keypair is registered it will be able to perform a few different actions such as request endorsements. In the future we will restrict other actions based on how endorsed a new account is. A problem with this relatively open model is Sybil attacks; where malactors register several keypairs and endorse each other to build a competing identity graph. One solution we may use to reduce this risk is to anchor our trust around emails with .edu and have some accounts verify email access. In the near future we would like to integrate the work of HyperLedger's Indy to leverage their robust digital identity framework. 
Pilot work on Hyperledger Sawtooth
We have deployed HyperLedger's Sawtooth infrastructure on AWS. One of the advantages of Sawtooth is it's integrated seth (Sawtooth ethereum virtual machine) transaction processor family which lets us port many of the written smart contracts to our free execution pipeline(in gas token terms). For the OpenClimate collabathon we encourage running your own instance and also interacting with ours.
 The other issue with data consensus is the several differing accounting methods. The anchors of our data accountability pipeline will be roll ups into ISO-3166 numerical codes. We hope to encourage a voluntary distributed approach to developing consensus around data by allowing our collaborators to submit records for schemas(file types), algorithms(executables for reproducibility), dApps(executables for interactions and data visualization), and of course datasets that are formatted into the aforementioned schemas. 
Most of these records will be simple structures for showing who has made these record attributions and who has verified the same data. The namespace for records in the sawtooth platform can be used to prevent duplicate records and have identical results simply additionally signed. Eventually we'll have datasets generated from different algorithms that can each have confidence scores, differing points of view from the identity graph, and cradle to grave accountability. This consensus is required to build higher abstractions such as financial instruments.
​
One of our weak areas currently is integrating the Transaction Processors and configuring sawtooth for our purposes. If you are familiar with these technologies please contact the datathon prompt host, Steven Ettinger, via discord.
Our First Transaction Processor (Smart Contract)
Have a look at the HyperLedger Sawtooth source documents.​
First we need a way to construct the payload that will trigger our process. These are built by the client and signed locally. Once signed our processor can validate the transaction to determine if state should change.
er_payload.js
1
'use strict'
2
​
3
const { InvalidTransaction } = require('sawtooth-sdk/processor/exceptions')
4
​
5
class EmRePayload {
6
    constructor(name, action, desc, addr) {
7
        this.name = name
8
        this.action = action
9
        this.desc = desc
10
        this.addr = addr
11
    }
12
​
13
    static fromBytes(payload) {
14
        payload = payload.toString().split(/,(?=(?:[^"]"[^"]")[^"]$)/gm) //allows nesting stringified objects
15
        if (payload.length === 4) {
16
            let emrePayload = new EmRePayload(payload[0], payload[1], payload[2], payload[3])
17
            if (!emrePayload.name) {
18
                throw new InvalidTransaction('Name is required')
19
            }
20
            if (emrePayload.name.indexOf('|') !== -1) {
21
                throw new InvalidTransaction('Name cannot contain "|"')
22
            }
23
​
24
            if (!emrePayload.action) {
25
                throw new InvalidTransaction('Action is required')
26
            }
27
            return emrePayload
28
        } else if (payload.length === 2) {
29
            let emrePayload = new EmRePayload(payload[0], payload[1])
30
            if (!emrePayload.name) {
31
                throw new InvalidTransaction('Name is required')
32
            }
33
            if (emrePayload.name.indexOf('|') !== -1) {
34
                throw new InvalidTransaction('Name cannot contain "|"')
35
            }
36
​
37
            if (emrePayload.action !== 'veri' || emrePayload.action !== 'fix') {
38
                throw new InvalidTransaction('Invalid action or data')
39
            }
40
            return emrePayload
41
        } else if (payload.length === 3) {
42
            let emrePayload = new EmRePayload(payload[0], payload[1], payload[2])
43
            if (!emrePayload.name) {
44
                throw new InvalidTransaction('Name is required')
45
            }
46
            if (emrePayload.name.indexOf('|') !== -1) {
47
                throw new InvalidTransaction('Name cannot contain "|"')
48
            }
49
​
50
            if (emrePayload.action !== 'fix') {
51
                throw new InvalidTransaction('Invalid action or data')
52
            }
53
            return emrePayload
54
        } else {
55
            throw new InvalidTransaction('Invalid payload serialization')
56
        }
57
    }
58
}
59
​
60
module.exports = EmRePayload
Copied!
Now we need to build a way to store the information in the sawtooth state.
er_state.js
1
'use strict'
2
const stringify = require('json-stable-stringify')
3
const crypto = require('crypto')
4
​
5
class EmReState {
6
    constructor(context) {
7
        this.context = context
8
        this.addressCache = new Map([])
9
        this.timeout = 500 // Timeout in milliseconds
10
    }
11
​
12
    getdRec(name) {
13
        return this._loaddRecs(name).then((dRecs) => dRecs.get(name))
14
    }
15
​
16
    setdRec(name, dRec) {
17
        let address = _makeEmReAddress(name) //by choosing our namespace with different portions of data we can ensure no record duplication
18
​
19
        return this._loaddRecss(name).then((dRecs) => {
20
            dRecs.set(name, dRec)
21
            return dRecs
22
        }).then((dRecs) => {
23
            let data = _serialize(dRecs)
24
​
25
            this.addressCache.set(address, data)
26
            let entries = {
27
                [address]: data
28
            }
29
            return this.context.setState(entries, this.timeout)
30
        })
31
    }
32
​
33
    deletedRec(name) {
34
        let address = _makeEmReAddress(name)
35
        return this._loaddRecs(name).then((dRecs) => {
36
            dRecs.delete(name)
37
​
38
            if (dRecs.size === 0) {
39
                this.addressCache.set(address, null)
40
                return this.context.deleteState([address], this.timeout)
41
            } else {
42
                let data = _serialize(dRecs)
43
                this.addressCache.set(address, data)
44
                let entries = {
45
                    [address]: data
46
                }
47
                return this.context.setState(entries, this.timeout)
48
            }
49
        })
50
    }
51
​
52
    _loaddRecs(name) {
53
        let address = _makeEmReAddress(name)
54
        if (this.addressCache.has(address)) {
55
            if (this.addressCache.get(address) === null) {
56
                return Promise.resolve(new Map([]))
57
            } else {
58
                return Promise.resolve(_deserialize(this.addressCache.get(address)))
59
            }
60
        } else {
61
            return this.context.getState([address], this.timeout)
62
                .then((addressValues) => {
63
                    if (!addressValues[address].toString()) {
64
                        this.addressCache.set(address, null)
65
                        return new Map([])
66
                    } else {
67
                        let data = addressValues[address].toString()
68
                        this.addressCache.set(address, data)
69
                        return _deserialize(data)
70
                    }
71
                })
72
        }
73
    }
74
}
75
​
76
const _hash = (x) =>
77
    crypto.createHash('sha512').update(x).digest('hex').toLowerCase().substring(0, 64)
78
​
79
const EMRE_FAMILY = 'emre'
80
​
81
const EMRE_NAMESPACE = _hash(EMRE_FAMILY).substring(0, 6)
82
​
83
const _makeEmReAddress = (x) => EMRE_NAMESPACE + _hash(x)
84
​
85
module.exports = {
86
    EMRE_NAMESPACE,
87
    EMRE_FAMILY,
88
    EmReState
89
}
90
​
91
const _deserialize = (data) => {
92
    let dRecsIterable = data.split('|').map(x => x.split(/,(?=(?:[^"]"[^"]")[^"]$)/gm))
93
        .map(x => [x[0], { name: x[0], by: x[1], addr: x[2], veri: JSON.parse(x[3]), data: JSON.parse(x[4]) }])
94
    return new Map(dRecsIterable)
95
}
96
​
97
const _serialize = (dRecs) => {
98
    let dRecStrs = []
99
    for (let namedRec of dRecs) {
100
        let name = namedRec[0]
101
        let dRec = namedRec[1]
102
        dRecStrs.push([name, dRec.by, dRec.addr, stringify(dRec.veri), stringify(dRec.data)].join(','))
103
    }
104
​
105
    return Buffer.from(dRecStrs.join('|'))
106
}
Copied!
The actual logic to handle transactions fits in the handler file. It generally chooses its functions by the action in the payload.
er_handler.js
1
'use strict'
2
​
3
const EmRePayload = require('./er_payload')
4
​
5
const { EMRE_NAMESPACE, EMRE_FAMILY, EmReState } = require('./er_state')
6
​
7
const { TransactionHandler } = require('sawtooth-sdk/processor/handler')
8
const { InvalidTransaction } = require('sawtooth-sdk/processor/exceptions')
9
​
10
​
11
class EMREHandler extends TransactionHandler {
12
    constructor() {
13
        super(EMRE_FAMILY, ['1.0'], [EMRE_NAMESPACE])
14
    }
15
​
16
    apply(transactionProcessRequest, context) {
17
        let payload = EmRePayload.fromBytes(transactionProcessRequest.payload)
18
        let emreState = new EmReState(context)
19
        let header = transactionProcessRequest.header
20
        let by = header.signerPublicKey
21
​
22
        if (payload.action === 'create') {
23
            return emreState.getdRec(payload.name)
24
                .then((dRec) => {
25
                    if (dRec !== undefined) {
26
                        throw new InvalidTransaction('Invalid Action: dRec already exists.')
27
                    }
28
                    let Allowed = isAllowed(payload.desc)
29
​
30
                    function isAllowed(str) {
31
                        return /[ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\,\.\-_\^\@\$\%\*\(\)\=\+\/]/.test(str);
32
                    }
33
                    if (!Allowed) {
34
                        throw new InvalidTransaction('Invalid Description: Only "Aa-Zz-_,.^@$%+/=" allowed in description.')
35
                    }
36
                    let createddRec = { //addressing records by their contents can ensure everybody signs the same records.
37
                        name: payload.desc.country + payload.desc.last_reported_year + payload.desc.last_reported_mmtco2e + payload.desc.gas_type,
38
                        addr: payload.addr,
39
                        by: by,
40
                        veri: [],
41
                        data: {
42
                            t: payload.desc.gas_type,
43
                            s: payload.desc.sector,
44
                            y: payload.desc.last_reported_year,
45
                            r: payload.desc.last_reported_mmtco2e,
46
                            c: payload.desc.country,
47
                            d: payload.desc.data_source,
48
                            p: payload.desc.data_reporter_public_key,
49
                            i: payload.desc.ipfs_identity,
50
                        }
51
                    }
52
​
53
​
54
                    return emreState.setdRec(payload.name, createddRec)
55
                })
56
        } else if (payload.action === 'veri') {
57
            return emreState.getdRec(payload.name)
58
                .then((dRec) => {
59
                    if (dRec === undefined) {
60
                        throw new InvalidTransaction(
61
                            'Invalid Action: Verify requires an existing dRec.'
62
                        )
63
                    }
64
                    let veri = false
65
                    for (i = 0; i < dRec.veri.length; i++) {
66
                        if (by === dRec.veri[i][0]) {
67
                            veri = true
68
                        }
69
                    }
70
                    if (by !== dRec.by && !veri) {
71
                        dRec.veri.push([by, 0])
72
                    }
73
​
74
                    return emreState.setdRec(payload.name, dRec)
75
                })
76
        } else if (payload.action === 'fix') {
77
            return emreState.getdRec(payload.name)
78
                .then((dRec) => {
79
                    if (dRec === undefined) {
80
                        throw new InvalidTransaction(
81
                            'Invalid Action: Verify requires an existing dRec.'
82
                        )
83
                    }
84
                    let veri = false
85
                    for (i = 0; i < dRec.veri.length; i++) {
86
                        if (by === dRec.veri[i][0]) {
87
                            veri = true
88
                        }
89
                    }
90
                    if (by !== dRec.by && !veri) {
91
​
92
                        dRec.veri.push([by, payload.desc])
93
                    }
94
​
95
                    return emreState.setdRec(payload.name, dRec)
96
                })
97
        } else if (payload.action === 'delete') {
98
            return emreState.getdRec(payload.name)
99
                .then((dRec) => {
100
                    if (dRec === undefined) {
101
                        throw new InvalidTransaction(
102
                            `No dRec exists with name ${payload.name}: unable to delete`)
103
                    } else if (by == dRec.by) {
104
                        return emreState.deletedRec(payload.name)
105
                    } else {
106
                        arr = drec.veri,
107
                            index = null
108
                        for (i = 0; i < arr.length; i++) {
109
                            if (arr[i][0] == by) {
110
                                index = i
111
                                break;
112
                            }
113
                        }
114
                        if (index > -1) {
115
                            dRec.veri.splice(index, 1)
116
                            return emreState.setdRec(payload.name, dRec)
117
                        } else {
118
                            throw new InvalidTransaction(
119
                                `You don't have permission to do that.`)
120
                        }
121
                    }
122
                })
123
        } else {
124
            throw new InvalidTransaction(
125
                `Action must be create, delete, or veri not ${payload.action}`
126
            )
127
        }
128
    }
129
}
130
​
131
module.exports = EMREHandler
Copied!
Finally a package.json and index file are built to handle to connections from sawtooth:
package.json
1
{
2
    "name": "drec_javascript",
3
    "version": "1.0.0",
4
    "description": "An implementation of the drec transaction family using the sawtooth JS sdk",
5
    "main": "index.js",
6
    "scripts": {
7
        "test": "echo \"Error: no test specified\" && exit 1",
8
        "start": "node index.js"
9
    },
10
    "author": "",
11
    "license": "Apache-2.0",
12
    "dependencies": {
13
        "cbor": "^3.0.0",
14
        "json-stable-stringify": "^1.0.1",
15
        "sawtooth-sdk": "file:./../../javascript"
16
    }
17
}
Copied!
index.js
1
'use strict'
2
​
3
const { TransactionProcessor } = require('sawtooth-sdk/processor')
4
const EMREHandler = require('./er_handler')
5
​
6
if (process.argv.length < 3) {
7
    console.log('missing a validator address')
8
    process.exit(1)
9
}
10
​
11
const address = process.argv[2]
12
​
13
const transactionProcessor = new TransactionProcessor(address)
14
​
15
transactionProcessor.addHandler(new EMREHandler())
16
​
17
transactionProcessor.start()
Copied!
This should form the entirety of the transaction handler. As you may be able to see to meet all of our requirements for data consensus this first approach won't quite fit all the wickets. 
First demo development efforts
Technical Considerations
Last modified 1yr ago
Copy link
Contents
The Data (Consensus) Challenge on blockchains
Pilot work on Hyperledger Sawtooth
Our First Transaction Processor (Smart Contract)