Location of config.yaml¶

The base yamjam config file is located ~/.yamjam/config.yaml

• on windows:

  c:\documents and settings\[username]\.yamjam\config.yaml

• on unix:

  ~/.yamjam/config.yaml

Then yamjam looks for possible overrides/additions in the current working directory of the APP for a file named config.yaml (if it exists) This would be the project config.yaml

You can override the config file(s) when you call the function by specifying a different one explicitly. i.e.

cfg = yamjam('/path/to/file/file')

By default YamJam looks for two config.yaml files, the base and then a project specific one. You may specify one or more files. Each file path listed should be separated by semi-colons(;) NOTE: if you specify files, YamJam then only looks for them and not the default base and project files.

cfg = yamjam('/path/to/file/file1;/other/path/file2')

Django Specific¶

In a Django environment, placing a config.yaml in the same directory as your settings.py file is the way to go, if needed. yamjam will look for the base config in ~/.yamjam/config.yaml and a project one in the directory where your settings.py file is located, by default.

settings.py

from YamJam import yamjam
...

CFG = yamjam()['myproject']

...

Then your config information from yamjam will be available any place you import settings.

from django.conf import settings

settings.CFG            # your yamjam values

Base config only¶

The vast majority of use cases can be accomplished with just a base config located at ~/.yamjam/config.yaml

Be hesitant to implement a project level config.yaml until you hit an edge case that requires it. This is the primary way of using YamJam. The project level config.yaml and user specified locations cause as many problems as the solve and or only applicable to a small set of edge cases. You will be best served by not straying from the KISS principle.

Structure of config.yaml¶

config.yaml is a straight up yaml file. See http://pyyaml.org/ and http://www.yaml.org/ for advanced usage. However, if you write python you know what you need to write decent yaml. White space is relevant and that is a good thing! What makes Python a joy to read is the same for yaml.

Single YAML document. A yaml file can contain multiple documents. YamJam will only evaluate the first document. Use namespace in the document to logically divide setting for multiple projects.

By default, Yamjam uses yaml’s safe_load directive to read the config file. This is for safety and sanity. If you find yourself wanting to store a custom python object in your config.yaml, you’ll want to re-evaluate your storage. We do leave the keys in the ignition and you can override .safe_load with .load by calling yamjam with the safe=False parameter. Missing toes from foot shooting is always the users responsibility.

Example¶

if you edit your ~/.yamjam/config.yaml file to include

myproject:
    key1: value
    key2: 42

It is returned from yamjam() as a dictionary.

{'myproject':
    {'key1': 'value',
     'key2': 42
    }
}

to get the value of key2, under myproject

>>> yamjam()['myproject']['key2']
42

Ok, now you know how to set up a dictionary in yaml, here is how you set up a list for a value.

edit config.yaml

myproject:
    key1: value
    key2: [42, 21, 7]

Now the value of key2 will return a list with 3 integers.

>>> yamjam()['myproject']['key2']
[42, 21, 7]

Now you are truly dangerous. You can create a dictionary of dictionaries that can contain more dictionaries, lists, integers, floats, strings, etc.

In fact yaml and pyyaml can do some wonderfully advanced stuff. But, always tend to the simplest form that will get the job done, think DRY KISS.

yjlint - linting for your config.yaml¶

When you installed YamJam, it created a console script yjlint that you can run from the command line to check your config.yaml files. To lint your config file:

yjlint ~/.yamjam/config.yaml

This will report any problems with the YAML in a config.yaml file.

If there is a YAML formatting error with any of your specified files, yamjam() will raise a YAMLError. You can of course, wrap your call to yamjam() with a try / except block.

from YamJam import yamjam, YAMLError
try:
    cfg = yamjam()
Except YAMLError:
    die_gracefully()

Use yjlint to make sure the YAML in your config is well formatted.

Logical Alignment¶

Since your config.yaml can store settings for more than one project and it almost always will, the standard use case is to create a section named for the project that uses it. i.e. If you have a Django project named ‘myproject’ then create a section in your config.yaml named myproject.

myproject:
    django-secret-key: my-secret-key

Ok, now you will need database connection information. So let’s add it.

myproject:
    django-secret-key: my-secret-key
    database:
        engine: django.db.backends.postgresql_psycopg2
        name: mydatabase
        user: mydatabaseuser
        password: mypassword
        host: 127.0.0.1
        port: 5432

Now let’s create config settings for our next-project

myproject:
    django-secret-key: my-secret-key
    database:
        engine: django.db.backends.postgresql_psycopg2
        name: mydatabase
        user: mydatabaseuser
        password: mypassword
        host: 127.0.0.1
        port: 5432

next-project:
    django-secret-key: next-project-secret-key
    database:
        engine: django.db.backends.postgresql_psycopg2
        name: mynextdatabase
        user: mynextdatabaseuser
        password: mynextpassword
        host: 127.0.0.1
        port: 5432

Ok, let’s access the django-secret-key for next-project

>>> yamjam()['next-project']['django-secret-key']
next-project-secret-key

So, now you can start factoring out your sensitive data from your app. One more thing, in case it is not obvious, you can reduce your typing by caching just the config information you need to access.

>>> cfg = yamjam()['next-project']
>>> cfg['django-secret-key']
next-project-secret-key

That will keep your code looking clean and also, you just cached the results of the yamjam() call. Further accesses to cfg now happen at break-neck python speed.

Each time your call yamjam() it re-reads the config files. Sometimes this is desired, sometimes not but as a programmer you have complete and simple control over how things happen.

Merging Config Data¶

When more than one config.yaml is processed, base and project are merged using the following 2 rules.

If the value being merged is not a dictionary then the base value is replaced by the project.

{‘key1’: val1} merged with {‘key1’: val2} results in {‘key1’: val2}

If the value being merged is a dictionary, then it is merged.

{‘key1’: {‘foo’: 2}} merged with {‘key1’: {‘bar’: 3}} results in {‘key1’: {‘foo’: 2, ‘bar’: 3}}

and

{‘key1’: {‘foo’: 2}} merged with {} results in {‘key1’: {‘foo’: 2}}

This way the project config only specified overrides and additions to the base config. See the tests for test_merge.py for more examples. But the net result of the merge should not surprise you and allow for you to supply the minimal amount of project config data needed to modify the base config.

Django Scenario - Sensitive Data¶

settings.py contains a number of configuration settings that should be kept secret/private and not be included in commits to source code control. So you either don’t check in your settings.py file (which creates its own set of problems) or your roll your own system. YamJam takes care of the tedious for you.

from YamJam import yamjam
...
DJANGO_SECRET_KEY = yamjam()['myproject']['django_secret_key']
...

myproject:
    django_secret_key: my-secret-key-value

Django Scenario - Differing Settings for Development/ Staging/ and Production¶

Your dev system uses a local data store for on the go development and to keep any migrations or bad data out of your staging and/or production environments. This means you need a different settings.py for each environment. Ok, there is a problem. Wouldn’t you prefer one settings.py file with the specialized settings handled without writing lots of if/else statements? Enter YamJam

from YamJam import yamjam
...
dbcfg = yamjam()['myproject']['database']

DATABASES = {
    'default': {
        'ENGINE':dbcfg['engine'],
        'NAME': dbcfg['name'],
        'USER': dbcfg['user'],
        'PASSWORD': dbcfg['password'],
        'HOST': dbcfg['host'],
        'PORT': dbcfg['port'],
    }
}

myproject:
    django_secret_key: my-secret-key-value
    database:
      engine: django.db.backends.postgresql_psycopg2
      name: mydatabase
      user: mydatabaseuser
      password: mypassword
      host: 127.0.0.1
      port: 5432

myproject:
    django_secret_key: my-secret-key-value
    database:
      engine: django.db.backends.postgresql_psycopg2
      name: mydatabase
      user: mydatabaseuser
      password: mypassword
      host: 10.0.0.24
      port: 5432

Multiple apps that access common resources¶

Do you ever write apps that integrate with other resources? Do you write additional apps that also integrate or report on those resources? If you find yourself not following the Don’t Repeat Yourself (DRY) conventions for resource information then YamJam is your friend.

• App1 access the corporate accounting db and the mail server and your no-sql data to automate reporting to vendors or customers.
• App2 access the corporate accounting db and your no-sql data and your fax system to generate POs.

Advanced DRY usage¶

The following is not specific to YamJam as it is in the YAML specs.

YAML allows for Alias indicators which allow for the following example:

bill-to: &id001
    given  : Monty
    family : Python
ship-to: *id001

which would return the following:

{'bill-to': {'given': 'Monty', 'family': 'Python'}, 'ship-to': {'given': 'Monty', 'family': 'Python'}}

This is helpful when you wish to refactor a key for a setting but you have other apps still relying on the old key. Until you can refactor those apps to use the new key, you can use an Alias to keep your information DRY

Why yaml for persistence?¶

All config dialects have their warts, some have more than others. However, YAML, in our opinion, is the most pythonic of the bunch. White space is relevant in yaml just like Python. Also, the visual look of yaml for dictionaries and lists is very congruous to Python.

key: value
adict:
    key: value
    key2: value2
alist: [1, 2, 3, 4]

How do I write to my config?¶

The short answer, is with your editor. YamJam is designed for change infrequently, read frequently scenarios. Your Django apps don’t write to settings.py, we don’t write to disk either.

Security¶

YamJam data is secured in the same method your .ssh directory is secured, chmod 700. It doesn’t support encryption intrinsically. It can return an encrypted value your have put there but that is up to your app. So follow the install instructions and chmod 700 your ~/.yamjam directory.

What is so special about YamJam, I could roll my own¶

Not a thing is technically special about YamJam. Yes, it’s got a decent mergeDict routine. The problem with rolling your own is the same type of problem as rolling your own templating system. Why not just pip install yamjam instead of maintaining the code yourself. That is the thing about YamJam, it’s easy to install. It takes the mundane task and rolls it up so you can write more exciting code. Just for the mere fact you can simply install it and then start factoring out sensitive and installation dependent data that has leaked into your repos.

What’s up with the name, YamJam?¶

One day I was sitting there, dealing with an issue of config differences between dev and production and I said to myself, wtf? Why can’t I just take my config settings and jam it in a yaml file? Hence, YamJam.

Code¶

1. Fork the repository on Bitbucket .
2. Make a virtualenv, clone the repos, install the deps from pip install -r requirements-dev.txt
3. Write any new tests needed and ensure existing tests continue to pass without modification.

1. Setup CI testing on drone.io for your Fork. See current script .

4. Ensure that your name is added to the end of the Authors file using the format Name <email@domain.com> (url), where the (url) portion is optional.
5. Submit a Pull Request to the project on Bitbucket.

Bug Reports¶

If you encounter a bug or some surprising behavior, please file an issue on our tracker

Get the Word Out¶

• Tell a friend
• email a list
• blog about about it
• give a talk to your local users group

Let’s face it, a config system is not that exciting (unless you get it wrong - Windows Registry). But just like templating, everyone needs it. You roll your own, you try ton’s of if/then and environment detection but it always ends up lacking. YamJam makes it sane and installable from pypi.

Now instead of spending attention cycles on your own boring implementation, you can spend your time on interesting code and let us take of the mundane.

Full Release¶

1. Merge in outstanding Pull requests
2. Ensure test suite pass / drone.io

3. run release.sh –dry-run and ensure all looks well
4. run release.sh and make sure to push any outstanding commits to master
5. create a new empty environment and pip install /localpath/yamjam-x.y.z.gz
6. copy runtests.py to the environment created in previous step and run it
7. upload package to pypi, twine upload sdist/*
8. verify new dist is available on pypi
9. Be Happy

Build Dist Only¶

python setup.py sdist

Install and test dist locally¶

1. Build Dist Only
2. Create clean environment
3. install local dist file in clean environment

pip install /localpath/to/dist/yamjam-x.y.z.gz

4. copy runtests.py to clean environment and run it
5. remove testing environment

v0.2.0¶

Planned...

• Deprecate and remove mergeDicts

v0.1.8¶

Planned...

• DeprecationWarning for mergeDicts

v0.1.7¶

In Progress..

• refactored code from module file to a proper package layout
• reimplement mergeDicts as merge_dicts
  • PendingDeprecationWarning for mergeDicts function
  • added tests for PendingDep warning and refactored existing tests to use merge_dicts
  • expanded test suite
• pep8 clean up of code
• add Change Log to documentation
• created a linting tool for config file, yjlint
• moved to yaml.safe_load, added option to allow old style, safe=False

v0.1.6¶

Current Release (released Apr 2014)

• migrated from code.google.com to bitbucket
• set up continuous integration
  • drone.io
  • tox
• implemented testing
  • py.test
  • tested explicitly on python 27, 32, 33, 34
  • anecdotally believed to run without issue on python 26
• wrote documentation, hosted on readthedocs.org
• started using twine to upload to pypi

v0.1.3 - v0.1.5¶

Did battle with long_description and pypi

v0.1.2¶

Previous Version (released Oct 2009)

• code hosted on code.google.com
• package hosted on code.google.com
• supported current python 26

v0.1.0 - v.0.1.1¶

(released Mar 2009)

• alpha-1 and alpha-2 releases