Rust-Joystream nicaea: Joystream — A user governed video platform

icon
Latest Release: nicaea

Joystream Build Status

This is the main code reposity for all joystream software. It will house the substrate chain project, the full node and runtime and all reusable substrate runtime modules that make up the joystream runtime. In addition to all front-end apps and infrastructure servers necessary for operating the network.

The repository is currently just a cargo workspace, but eventually will also contain yarn workspaces, and possibly other project type workspaces.

Build Status

Development Development Branch Build Status

More detailed build history on Travis CI

Overview

The Joystream network builds on a pre-release version of substrate v2.0 and adds additional functionality to support the various roles that can be entered into on the platform.

Validator

 Nodes for Joystream

Joystream node is the main server application that connects to the network, synchronizes the blockchain with other nodes and produces blocks if configured as a validator node.

To setup a full node and validator review the advanced guide from the helpdesk.

Pre-built Binaries

The latest pre-built binaries can be downloaded from the releases page.

Building from source

Clone the repository and install build tools:

git clone https://github.com/Joystream/joystream.git

cd joystream/

./setup.sh

Building

cargo build --release

Running a public node on the Rome testnet

Run the node and connect to the public testnet.

cargo run --release -- --chain ./rome-tesnet.json

The rome-testnet.json chain file can be obtained from the releases page

Installing a release build

This will install the executable joystream-node to your ~/.cargo/bin folder, which you would normally have in your $PATH environment.

cargo install joystream-node --path node/

Now you can run

joystream-node --chain rome-testnet.json

Local development

This will build and run a fresh new local development chain purging existing one:

./scripts/run-dev-chain.sh

Unit tests

cargo test

Network tests

./scripts/run-test-chain.sh
yarn test

To run the integration tests with a different chain, you can omit step running the local development chain and set the node URL using NODE_URL environment variable. Proposal grace periods should be set to 0, otherwise proposal network tests will fail.

Rome-Constantinople migration network test

Ensure Rome node is up and running, and node URL is set using NODE_URL environment variable (default value is localhost:9944).

yarn test-migration

Joystream Runtime

Joystream Runtime

The runtime is the code that defines the consensus rules of the Joystream protocol. It is compiled to WASM and lives on chain. Joystream node execute the code's logic to validate transactions and blocks on the blockchain.

When building joystream-node as described abot with cargo build --release, in addition to the joystream-node binary being built the WASM blob artifact is produced in:

target/release/wbuild/joystream-node-runtime/joystream_node_runtime.compact.wasm

Deployment

Deploying the compiled runtime on a live system can be done in one of two ways:

  1. Joystream runtime upgrade proposals which will be voted on by the council. When the Joystream platform is live, this will be the only way to upgrade the chain's runtime code.

  2. During development and testnet phases, we can send an extrinsic (transaction signed with the sudo key) invoking system::setCode(). This can be done either from the UI/extrinsics app, or directly with an admin script.

Versioning the runtime

Versioning of the runtime is set in runtime/src/lib.rs For detailed information about how to set correct version numbers when developing a new runtime, see this

Coding style

We use cargo-fmt to format the source code for consistency.

It should be available on your machine if you ran the setup.sh script, otherwise install it with rustup:

rustup component add rustfmt

Applying code formatting on all source files recursing subfolders:

cargo-fmt

Contributing

Please see our contributing guidlines for details on our code of conduct, and the process for submitting pull requests to us.

Authors

See also the list of CONTRIBUTORS who participated in this project.

License

This project is licensed under the GPLv3 License - see the LICENSE file for details

Acknowledgments

Thanks to the whole Parity Tech team for making substrate and helping on riot chat with tips, suggestions, tutorials and answering all our questions during development.

Comments

  • Configure Min/Max bounty reward value in Runtime
    Configure Min/Max bounty reward value in Runtime

    Jan 17, 2022

    Context During development we work on placeholder values. Currently Bounties min/max values are not defined for the release.

    To-do

    • [ ] Define params for bounties min/max values @bwhm
    • [ ] Update the values in the runtime config
    runtime bounty 
    Reply
  • DevOsp - node-network custom domain using ACM and CNAME
    DevOsp - node-network custom domain using ACM and CNAME

    Jan 18, 2022

    This PR adds support to add Custom Domain to our deployments

    Instructions

    • Go to AWS Console > AWS Certificate Manager
    • Create new Request and add your domain name Screenshot 2022-01-18 at 5 55 51 PM
    • Copy the CNAME details and add to your DNS management console Screenshot 2022-01-18 at 5 49 12 PM
    • Copy the ARN of the certificate for future use (e.g. arn:aws:acm:us-east-1:882345074216:certificate/4ad63435-7145-413b-af70-3a19ed66ea4a)
    • Go to node-network and set the config paramerters
    • Set acmCertificateARN to the value from above
    • Run pulumi up -y
    • After deployment, copy the endpoint1 output Screenshot 2022-01-18 at 5 49 26 PM
    • Set the CNAME record in the DNS management console to the Load Balancer endpoint value Screenshot 2022-01-18 at 5 49 08 PM
    • You can now access the endpoint in a secure way Screenshot 2022-01-18 at 5 51 03 PM
    Reply
  • Giza feature new content features
    Giza feature new content features

    Jan 18, 2022

    channel payouts:

    • Reference: #2365.
    • on chain variable used for this features, namely Commitment, MaxRewardAllowed, MinCashoutAllowed originally updated by root now they can be updated by Lead.
    • Reward for the channel is minted directly into the reward_account

    video comments

    • Reference: #2424, #2492

    Reviewer: @shamil-gadelshin Adding also @Lezek123 as a reviewer (so at least he stays updated in case type changes)

    Reply
  • Runtime: Bounties Enhancement
    Runtime: Bounties Enhancement

    Jan 18, 2022

    null

                                                                                                                                                                                                            enhancement Epic 
    Reply
  • Olympia - Fix the OracleJudgment type
    Olympia - Fix the OracleJudgment type

    Jan 18, 2022

    The way OracleJudgment was defined:

    export class OracleJudgment extends JoyEnum({
      Winner: OracleJudgment_Winner,
      Rejected: Null,
    }) {}
    

    Is not usable to map OracleJudgmentSubmitted each entry status. Plus in the runtime code OracleJudgment is:

    pub type OracleJudgment<EntryId, Balance> = BTreeMap<EntryId, OracleWorkEntryJudgment<Balance>>;
    
    Reply
  • Ops and R&D Tasks Q1 2022
    Ops and R&D Tasks Q1 2022

    Jan 18, 2022

    Context

    This is a tracking/ placeholder temp issue to triage the actions and tasks from the latest retrospective for the network team held on 13th Jan 22.

    Link to the Miro Board >

    Tasks

    1️⃣ Critical to start right now

    @all

    • [ ] Check if in the current PRs we have made progress on assigning more than 1 person for review. Action point was "Dual reviews of API interfaces PRs."

    @Lezek123

    • [ ] Review Old vs New Schema for Memberships
    • [ ] Review Old vs New Schema for Faucet

    2️⃣ Part of Olympia scope @all

    • [ ] Log changes spanning all key updates since designs via GH file called "Changelog".

    @Lezek123 @mnaamani

    • [ ] Do CI tasks earlier

    3️⃣ After Olympia

    • [ ] #3037
    • [ ] #1593
    • [ ] #1595
    • [ ] #3036
    • [ ] #3035
    tech-debt devops Epic 
    Reply
  • Comments to videos
    Comments to videos

    Jun 21, 2021

    Proposal

    T.O.C.

    1. Goal
    2. Post structure
      1. Post
      2. Replies / Comments
    3. Storage Elements
      1. Posts
      2. Replies
    4. Extrinsics 1. Create Post 2. Create Reply 3. Edit Post 4. Edit Reply 5. Delete Post 6. Delete Reply 7. React to Post
    5. Questions & precisations

    Target Branch: new_channel_features

    Module: content

    Goal

    Enrich each video with features for express a rating and comments, something similar to a Youtube/Vimeo page. Basic idea A video should start a thread post (since in a thread all the posted elements tends to be threated as simple posts/replies). Instead what we want is a post that starts with a video and thereafter there are only replies (to the post or to other replies). Effectively it should be something along the lines of:

    Post structure

    Basic idea: A video that lives in a channel should start a post. Once the post is set up only text replies are allowed by post owner and other users We also want the video to be in the channel owned by the post owner

    Post

    We want the video to be in the channel owner by the post owner

      pub struct Post<ChannelOwner,
    		  Hash,
    		  Balance,
    		  BlockNumber,
    		  ReplyId, 
    		  VideoCategoryId
    		  > {
          /// Id of thread to which this post corresponds.
    n
          /// Hash of current text
          pub title_hash: Hash,
    
          /// Author of post.
          pub author: ChannelOwner,
    
          /// Cleanup pay off
          pub cleanup_pay_off: Balance,
    
          /// When it was created or last edited
          pub last_edited: BlockNumber,
    
          /// Overall replies counter
          pub replies_count: ReplyId,
    
          /// video associated to the post (instead of the body hash as in the blog module)
          pub video: VideoId,
    
          /// category for this video
          pub category: CategoryId
      }
    

    The category field allows to use the same structure for moderation (Moderators, Lead …) as in the forum module.

    Replies / Comments

    Reply is analogous to a Youtube comment, so every post (video) will have zero or more replies, and also a reply can have zero or more reply to it. It makes sense therefore to identify as a Parent a Post | Reply that can have zero or more replies to it.

    pub enum ParentId<ReplyId, PostId: Default> {
        Reply(ReplyId),
        Post(PostId),
    }
    

    And a reply struct will be something like

    pub struct Reply<BlockNumber,
    		 ReplyId,
    		 PostId,
    		 UserId,
    		 Balance,
    		 Hash> {
        /// Reply text hash
        text_hash: Hash,
        /// Associated with reply owner
        owner: UserId, 
        /// Reply`s parent id
        parent_id: ParentId<T::ReplyId, PostId>,
        /// Pay off by deleting post
        cleanup_pay_off: Balance,
        /// Last time reply was edited
        last_edited: BlockNumber,
    }
    

    Storage Elements

    Posts

    Stored by means of a single map

    pub PostById get(fn post_by_id) config() : map
        hasher(blake2_128_concat) PostId => Post;
    

    Replies

    Stored by means of a double map (PostId,

    ReplyById get (fn reply_by_id): double_map hasher(blake2_128_concat) PostId, hasher(blake2_128_concat) ReplyId => Reply>;
    

    Extrinsics

    Create Post

    We have to make sure that the post owner is the owner of the channel that the video belongs to.

      pub fn create_post(origin,
    		     title: Hash,
    		     video: VideoId,
    		     category: CategoryId
      ) -> DispatchResult 
    {
        ...
    	let post_owner = <ChannelById>::get(
    	    <VideoById>::get(video)
    		.in_channel)
    	.channel_owner;
        ...
    }
    

    Create Reply

    pub fn create_reply(
        origin,
        owner: UserId,
        post_id: PostId,
        reply_id: Option<ReplyId>,
        text: Hash,
    ) -> DispatchResult {
    

    reply_id will be different from None in case of nested replies.

    Edit Post

    pub fn edit_post(
        origin,
        post_id: PostId,
        new_title: Option<Hash>
        new_video: Option<VideoId>
    ) -> DispatchResult {
    

    We must make sure that origin is signed by the post owner. If new_title (resp. new_video) are none title (video) stays the same.

    Edit Reply

    pub fn edit_reply(
        origin,
        owner: UserId,
        post_id: PostId,
        reply_id: T::ReplyId,
        new_text: Hash
    ) -> DispatchResult
    

    Replace the reply [ReplyId,PostId] (if it exists) in the ReplyById map using the new parameters, after ensuring that origin is signed by the owner of the reply in question.

    Delete Post

    pub fn delete_post(
        origin,
        owner: ChannelOwner,
        post: PostId,
    ) -> DispatchResult
    

    After making sure that origin is signed by the post owner, delete the post and possibly all the replies to it.

    Delete Reply

    pub fn delete_replies(
        origin,
        owner: UserId,
        reply: ReplyId,
    ) -> DispatchResult
    

    After ensuring that origin is signed by the owner of reply, delete the reply in question and possibly all the replies to it.

    React to Post

    fn react_post(origin,
    	      post: PostId,
    	      react: PostReactionId
    ) -> DispatchResult
    

    Add a reaction to the post

    Questions & precisations

    1. Should I write the code in a new file and in a new pallet
    2. When I delete a reply should also delete all the replies to it? Same thing when deleting a post?
    3. I used UserId to represent a reply owner different from ChannelOwner on purpouse: you can reply even if you don't own a channel. Effectively UserId will be common::MemberId (that is the member of the system).
    4. Should I add a reaction_score field to the Post data structure in order to keep track of the reaction? It appears that in the forum module a reaction causes only an Event deposit
    Reply
  • Fee model verification methodology
    Fee model verification methodology

    Feb 1, 2021

    Take a benchmarked runtime and any other required information as given,

    1. For a specific proposed weight price w (s/JOY) and byte price f (bytes/JOY), by what methodology to classify this combination as acceptable or not acceptable.
    2. For two such combinations(w_1,f_1) and (w_2,f_2), by what methodology can we order them to say (w_1,f_1) >=(w_2,f_2) in terms of desirability.

    Without this it seems hard to come up with suitable tools of analysis for determining such prices.

    question benchmarking 
    Reply
  • Minor fixes agreed upon for Giza
    Minor fixes agreed upon for Giza

    Sep 30, 2021

    • Drop data object id association in video: maybe_data_objects_id_set
    • Drop incrementing ChannelRecord::num_videos in update_video.
    • introduce atomic data object deletion in: for all of these, user provides objects: BTreeSet<DataObjectId>, for all each, they will check that deletion is feasible (by checking bag membership of all objects), and then this is done as first side-effect after mutation barrier.
      • update_video
      • update_channel
      • delete_video
      • delete_channel
    content directory giza 
    Reply
  • Pallets where there is no deposit charged for filling up the storage
    Pallets where there is no deposit charged for filling up the storage

    Feb 23, 2021

    Problem

    It was found during the audit that any forum user can use the forum's extrinsic create_thread to fill up the blockchain storage and exhaust the number of threads of a given category for a small fee since this requires no deposit.

    This problem exists in two forms across the runtime: Either we allow with no deposit filling up the storage of a given module with no bound or with bounds.

    In the bounded case, this would allow a denial of service attack where other users won't be able to use the storage.

    In the unbounded case, this'd allow to grow the storage maliciously.

    There are different permission levels that'd allow a user to abuse these extrinsics, either a lead or a worker/member.

    When a lead can fill up a bounded storage is not really a problem because there is no other user that it can be denied of service but I leave those cases in here just to keep them in mind.

    Pallets

    • forum

      • create_thread: Any forum user can create a thread, the thread storage for a given category is bounded.
      • add_post: Any forum user can create a post, the post storage for a given thread is bounded.
      • create_category: The lead can create categories with no cost, the category storage is bounded. (Same goes for sub-categories)
    • working_group

      • add_opening: A lead can use this extrinsic to create openings unbounded.
      • apply_on_opening: A working group member can apply multiple times, filling blockchain storage(when no stakes) unbounded.
      • add_opening + apply_on_opening + fill_opening: A lead can add an opening with no stakes, apply on it multiple times and fill it, and occupy completely the worker storage.
    • membership

      • add_staking_account_candidate: A user could use multiple accounts to fill up the blockchain storage, unbounded.
    • proposals_discussion

      • add_post: A member can add posts to any thread, unbounded.
    • storage/data_directory

      • add_content: A member can use it to fill up the storage unbounded.
    • content_directory

      • add_curator_group: A lead can add groups to fill up the storage unbounded.
      • add_curator_to_group: A lead can add curators to a group to fill it up bounded.
    • service_discovery:

      • set_ipns_id: Any worker can use it to fill up storage unbounded.
    • data_object_type_registry

      • register_data_object_type: A lead can use it to fill up storage unbounded.
    Reply
  • Giza Tracking Issue
    Giza Tracking Issue

    Jul 28, 2021

    Purpose

    The purpose of this issue to track overall progress on the storage system in the Giza release. This work is complex, in particular in the in that it introduce two new nodes, coordinated by a new large runtime module, and which have to interact in large numbers with each other. It will be critical to have the lead developers on this system to stay tightly synchronized, and keep other people depending on their work (e.g. the release manager, devops, etc.) updated on their progress.

    giza 
    Reply
  • Substrate coding conventions I
    Substrate coding conventions I

    May 6, 2019

    Add your suggestion as a comment!

    Background

    Our Substrate code base is starting to get more complicated, and it would be a benefit to harmonise the set of major conventions we follow, so as to follow good best practices, and make reviews more efficient. The goal of this issue is to accumulate suggestions over time, as replies, which we can turn into an eventual convention document. This document can further be turned into rules for our CI linter.

    Major questions that

    • How to decide what is its own module, vs. combining with existing module?
    • Modules deserve their own repo? every module?

    Initial suggestions

    • All maps must map to Option to avoid default construction behaviour of StorageMap from allowing us to be lazy about checking ::exists on the same map before lookup.

    • Always try to make a module easily reusable for another runtime, by:

      • If possible, always provide your own traits for your expectations on other modules, rather than relying on public traits, or traits in the module you are expecting to use.
      • Don't define and implement traits for your own module, see point above.
      • If possible, strive to implement the business logic of your module separately from the runtime module itself, in a substrate agnostic way.
    • Assert as many invariants as possible!

    Reply