Mirrors/servo
1
0
Fork 0
mirror of https://github.com/servo/servo.git synced 2025-06-12 18:34:39 +00:00
Code Issues Projects Releases Packages Wiki Activity

Update mach from gecko tree

Browse source
This commit is contained in:
James Graham 2015-07-01 10:53:23 +01:00
parent 9897125b34
commit f1641fde8f
18 changed files with 785 additions and 520 deletions
Download patch file Download diff file Expand all files Collapse all files

317
python/mach/README.rst
View file

@ -10,319 +10,4 @@ executable *driver* script (named whatever you want), and write mach
commands. When the *driver* is executed, mach dispatches to the
requested command handler automatically.
Features
========
On a high level, mach is similar to using argparse with subparsers (for
command handling). When you dig deeper, mach offers a number of
additional features:
Distributed command definitions
With optparse/argparse, you have to define your commands on a central
parser instance. With mach, you annotate your command methods with
decorators and mach finds and dispatches to them automatically.
Command categories
Mach commands can be grouped into categories when displayed in help.
This is currently not possible with argparse.
Logging management
Mach provides a facility for logging (both classical text and
structured) that is available to any command handler.
Settings files
Mach provides a facility for reading settings from an ini-like file
format.
Components
==========
Mach is conceptually composed of the following components:
core
The mach core is the core code powering mach. This is a Python package
that contains all the business logic that makes mach work. The mach
core is common to all mach deployments.
commands
These are what mach dispatches to. Commands are simply Python methods
registered as command names. The set of commands is unique to the
environment mach is deployed in.
driver
The *driver* is the entry-point to mach. It is simply an executable
script that loads the mach core, tells it where commands can be found,
then asks the mach core to handle the current request. The driver is
unique to the deployed environment. But, it's usually based on an
example from this source tree.
Project State
=============
mach was originally written as a command dispatching framework to aid
Firefox development. While the code is mostly generic, there are still
some pieces that closely tie it to Mozilla/Firefox. The goal is for
these to eventually be removed and replaced with generic features so
mach is suitable for anybody to use. Until then, mach may not be the
best fit for you.
Implementing Commands
---------------------
Mach commands are defined via Python decorators.
All the relevant decorators are defined in the *mach.decorators* module.
The important decorators are as follows:
CommandProvider
A class decorator that denotes that a class contains mach
commands. The decorator takes no arguments.
Command
A method decorator that denotes that the method should be called when
the specified command is requested. The decorator takes a command name
as its first argument and a number of additional arguments to
configure the behavior of the command.
CommandArgument
A method decorator that defines an argument to the command. Its
arguments are essentially proxied to ArgumentParser.add_argument()
Classes with the *@CommandProvider* decorator *must* have an *__init__*
method that accepts 1 or 2 arguments. If it accepts 2 arguments, the
2nd argument will be a *MachCommandContext* instance. This is just a named
tuple containing references to objects provided by the mach driver.
Here is a complete example::
from mach.decorators import (
CommandArgument,
CommandProvider,
Command,
)
@CommandProvider
class MyClass(object):
@Command('doit', help='Do ALL OF THE THINGS.')
@CommandArgument('--force', '-f', action='store_true',
help='Force doing it.')
def doit(self, force=False):
# Do stuff here.
When the module is loaded, the decorators tell mach about all handlers.
When mach runs, it takes the assembled metadata from these handlers and
hooks it up to the command line driver. Under the hood, arguments passed
to the decorators are being used to help mach parse command arguments,
formulate arguments to the methods, etc. See the documentation in the
*mach.base* module for more.
The Python modules defining mach commands do not need to live inside the
main mach source tree.
Conditionally Filtering Commands
--------------------------------
Sometimes it might only make sense to run a command given a certain
context. For example, running tests only makes sense if the product
they are testing has been built, and said build is available. To make
sure a command is only runnable from within a correct context, you can
define a series of conditions on the *Command* decorator.
A condition is simply a function that takes an instance of the
*CommandProvider* class as an argument, and returns True or False. If
any of the conditions defined on a command return False, the command
will not be runnable. The doc string of a condition function is used in
error messages, to explain why the command cannot currently be run.
Here is an example:
from mach.decorators import (
CommandProvider,
Command,
)
def build_available(cls):
"""The build needs to be available."""
return cls.build_path is not None
@CommandProvider
class MyClass(MachCommandBase):
def __init__(self, build_path=None):
self.build_path = build_path
@Command('run_tests', conditions=[build_available])
def run_tests(self):
# Do stuff here.
It is important to make sure that any state needed by the condition is
available to instances of the command provider.
By default all commands without any conditions applied will be runnable,
but it is possible to change this behaviour by setting *require_conditions*
to True:
m = mach.main.Mach()
m.require_conditions = True
Minimizing Code in Commands
---------------------------
Mach command modules, classes, and methods work best when they are
minimal dispatchers. The reason is import bloat. Currently, the mach
core needs to import every Python file potentially containing mach
commands for every command invocation. If you have dozens of commands or
commands in modules that import a lot of Python code, these imports
could slow mach down and waste memory.
It is thus recommended that mach modules, classes, and methods do as
little work as possible. Ideally the module should only import from
the *mach* package. If you need external modules, you should import them
from within the command method.
To keep code size small, the body of a command method should be limited
to:
1. Obtaining user input (parsing arguments, prompting, etc)
2. Calling into some other Python package
3. Formatting output
Of course, these recommendations can be ignored if you want to risk
slower performance.
In the future, the mach driver may cache the dispatching information or
have it intelligently loaded to facilitate lazy loading.
Logging
=======
Mach configures a built-in logging facility so commands can easily log
data.
What sets the logging facility apart from most loggers you've seen is
that it encourages structured logging. Instead of conventional logging
where simple strings are logged, the internal logging mechanism logs all
events with the following pieces of information:
* A string *action*
* A dict of log message fields
* A formatting string
Essentially, instead of assembling a human-readable string at
logging-time, you create an object holding all the pieces of data that
will constitute your logged event. For each unique type of logged event,
you assign an *action* name.
Depending on how logging is configured, your logged event could get
written a couple of different ways.
JSON Logging
------------
Where machines are the intended target of the logging data, a JSON
logger is configured. The JSON logger assembles an array consisting of
the following elements:
* Decimal wall clock time in seconds since UNIX epoch
* String *action* of message
* Object with structured message data
The JSON-serialized array is written to a configured file handle.
Consumers of this logging stream can just perform a readline() then feed
that into a JSON deserializer to reconstruct the original logged
message. They can key off the *action* element to determine how to
process individual events. There is no need to invent a parser.
Convenient, isn't it?
Logging for Humans
------------------
Where humans are the intended consumer of a log message, the structured
log message are converted to more human-friendly form. This is done by
utilizing the *formatting* string provided at log time. The logger
simply calls the *format* method of the formatting string, passing the
dict containing the message's fields.
When *mach* is used in a terminal that supports it, the logging facility
also supports terminal features such as colorization. This is done
automatically in the logging layer - there is no need to control this at
logging time.
In addition, messages intended for humans typically prepends every line
with the time passed since the application started.
Logging HOWTO
-------------
Structured logging piggybacks on top of Python's built-in logging
infrastructure provided by the *logging* package. We accomplish this by
taking advantage of *logging.Logger.log()*'s *extra* argument. To this
argument, we pass a dict with the fields *action* and *params*. These
are the string *action* and dict of message fields, respectively. The
formatting string is passed as the *msg* argument, like normal.
If you were logging to a logger directly, you would do something like:
logger.log(logging.INFO, 'My name is {name}',
extra={'action': 'my_name', 'params': {'name': 'Gregory'}})
The JSON logging would produce something like:
[1339985554.306338, "my_name", {"name": "Gregory"}]
Human logging would produce something like:
0.52 My name is Gregory
Since there is a lot of complexity using logger.log directly, it is
recommended to go through a wrapping layer that hides part of the
complexity for you. The easiest way to do this is by utilizing the
LoggingMixin:
import logging
from mach.mixin.logging import LoggingMixin
class MyClass(LoggingMixin):
def foo(self):
self.log(logging.INFO, 'foo_start', {'bar': True},
'Foo performed. Bar: {bar}')
Entry Points
============
It is possible to use setuptools' entry points to load commands
directly from python packages. A mach entry point is a function which
returns a list of files or directories containing mach command
providers. e.g.::
def list_providers():
providers = []
here = os.path.abspath(os.path.dirname(__file__))
for p in os.listdir(here):
if p.endswith('.py'):
providers.append(os.path.join(here, p))
return providers
See http://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins
for more information on creating an entry point. To search for entry
point plugins, you can call *load_commands_from_entry_point*. This
takes a single parameter called *group*. This is the name of the entry
point group to load and defaults to ``mach.providers``. e.g.::
mach.load_commands_from_entry_point("mach.external.providers")
Adding Global Arguments
=======================
Arguments to mach commands are usually command-specific. However,
mach ships with a handful of global arguments that apply to all
commands.
It is possible to extend the list of global arguments. In your
*mach driver*, simply call ``add_global_argument()`` on your
``mach.main.Mach`` instance. e.g.::
mach = mach.main.Mach(os.getcwd())
# Will allow --example to be specified on every mach command.
mach.add_global_argument('--example', action='store_true',
help='Demonstrate an example global argument.')
To learn more, read the docs in ``docs/``.

145
python/mach/docs/commands.rst Normal file
View file

@ -0,0 +1,145 @@
.. _mach_commands:
=====================
Implementing Commands
=====================
Mach commands are defined via Python decorators.
All the relevant decorators are defined in the *mach.decorators* module.
The important decorators are as follows:
:py:func:`CommandProvider <mach.decorators.CommandProvider>`
A class decorator that denotes that a class contains mach
commands. The decorator takes no arguments.
:py:func:`Command <mach.decorators.Command>`
A method decorator that denotes that the method should be called when
the specified command is requested. The decorator takes a command name
as its first argument and a number of additional arguments to
configure the behavior of the command.
:py:func:`CommandArgument <mach.decorators.CommandArgument>`
A method decorator that defines an argument to the command. Its
arguments are essentially proxied to ArgumentParser.add_argument()
:py:func:`SubCommand <mach.decorators.SubCommand>`
A method decorator that denotes that the method should be a
sub-command to an existing ``@Command``. The decorator takes the
parent command name as its first argument and the sub-command name
as its second argument.
``@CommandArgument`` can be used on ``@SubCommand`` instances just
like they can on ``@Command`` instances.
Classes with the ``@CommandProvider`` decorator **must** have an
``__init__`` method that accepts 1 or 2 arguments. If it accepts 2
arguments, the 2nd argument will be a
:py:class:`mach.base.CommandContext` instance.
Here is a complete example:
.. code-block:: python
from mach.decorators import (
CommandArgument,
CommandProvider,
Command,
)
@CommandProvider
class MyClass(object):
@Command('doit', help='Do ALL OF THE THINGS.')
@CommandArgument('--force', '-f', action='store_true',
help='Force doing it.')
def doit(self, force=False):
# Do stuff here.
When the module is loaded, the decorators tell mach about all handlers.
When mach runs, it takes the assembled metadata from these handlers and
hooks it up to the command line driver. Under the hood, arguments passed
to the decorators are being used to help mach parse command arguments,
formulate arguments to the methods, etc. See the documentation in the
:py:mod:`mach.base` module for more.
The Python modules defining mach commands do not need to live inside the
main mach source tree.
Conditionally Filtering Commands
================================
Sometimes it might only make sense to run a command given a certain
context. For example, running tests only makes sense if the product
they are testing has been built, and said build is available. To make
sure a command is only runnable from within a correct context, you can
define a series of conditions on the
:py:func:`Command <mach.decorators.Command>` decorator.
A condition is simply a function that takes an instance of the
:py:func:`mach.decorators.CommandProvider` class as an argument, and
returns ``True`` or ``False``. If any of the conditions defined on a
command return ``False``, the command will not be runnable. The
docstring of a condition function is used in error messages, to explain
why the command cannot currently be run.
Here is an example:
.. code-block:: python
from mach.decorators import (
CommandProvider,
Command,
)
def build_available(cls):
"""The build needs to be available."""
return cls.build_path is not None
@CommandProvider
class MyClass(MachCommandBase):
def __init__(self, build_path=None):
self.build_path = build_path
@Command('run_tests', conditions=[build_available])
def run_tests(self):
# Do stuff here.
It is important to make sure that any state needed by the condition is
available to instances of the command provider.
By default all commands without any conditions applied will be runnable,
but it is possible to change this behaviour by setting
``require_conditions`` to ``True``:
.. code-block:: python
m = mach.main.Mach()
m.require_conditions = True
Minimizing Code in Commands
===========================
Mach command modules, classes, and methods work best when they are
minimal dispatchers. The reason is import bloat. Currently, the mach
core needs to import every Python file potentially containing mach
commands for every command invocation. If you have dozens of commands or
commands in modules that import a lot of Python code, these imports
could slow mach down and waste memory.
It is thus recommended that mach modules, classes, and methods do as
little work as possible. Ideally the module should only import from
the :py:mod:`mach` package. If you need external modules, you should
import them from within the command method.
To keep code size small, the body of a command method should be limited
to:
1. Obtaining user input (parsing arguments, prompting, etc)
2. Calling into some other Python package
3. Formatting output
Of course, these recommendations can be ignored if you want to risk
slower performance.
In the future, the mach driver may cache the dispatching information or
have it intelligently loaded to facilitate lazy loading.

51
python/mach/docs/driver.rst Normal file
View file

@ -0,0 +1,51 @@
.. _mach_driver:
=======
Drivers
=======
Entry Points
============
It is possible to use setuptools' entry points to load commands
directly from python packages. A mach entry point is a function which
returns a list of files or directories containing mach command
providers. e.g.:
.. code-block:: python
def list_providers():
providers = []
here = os.path.abspath(os.path.dirname(__file__))
for p in os.listdir(here):
if p.endswith('.py'):
providers.append(os.path.join(here, p))
return providers
See http://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins
for more information on creating an entry point. To search for entry
point plugins, you can call
:py:meth:`mach.main.Mach.load_commands_from_entry_point`. e.g.:
.. code-block:: python
mach.load_commands_from_entry_point("mach.external.providers")
Adding Global Arguments
=======================
Arguments to mach commands are usually command-specific. However,
mach ships with a handful of global arguments that apply to all
commands.
It is possible to extend the list of global arguments. In your
*mach driver*, simply call
:py:meth:`mach.main.Mach.add_global_argument`. e.g.:
.. code-block:: python
mach = mach.main.Mach(os.getcwd())
# Will allow --example to be specified on every mach command.
mach.add_global_argument('--example', action='store_true',
help='Demonstrate an example global argument.')

74
python/mach/docs/index.rst Normal file
View file

@ -0,0 +1,74 @@
====
mach
====
Mach (German for *do*) is a generic command dispatcher for the command
line.
To use mach, you install the mach core (a Python package), create an
executable *driver* script (named whatever you want), and write mach
commands. When the *driver* is executed, mach dispatches to the
requested command handler automatically.
Features
========
On a high level, mach is similar to using argparse with subparsers (for
command handling). When you dig deeper, mach offers a number of
additional features:
Distributed command definitions
With optparse/argparse, you have to define your commands on a central
parser instance. With mach, you annotate your command methods with
decorators and mach finds and dispatches to them automatically.
Command categories
Mach commands can be grouped into categories when displayed in help.
This is currently not possible with argparse.
Logging management
Mach provides a facility for logging (both classical text and
structured) that is available to any command handler.
Settings files
Mach provides a facility for reading settings from an ini-like file
format.
Components
==========
Mach is conceptually composed of the following components:
core
The mach core is the core code powering mach. This is a Python package
that contains all the business logic that makes mach work. The mach
core is common to all mach deployments.
commands
These are what mach dispatches to. Commands are simply Python methods
registered as command names. The set of commands is unique to the
environment mach is deployed in.
driver
The *driver* is the entry-point to mach. It is simply an executable
script that loads the mach core, tells it where commands can be found,
then asks the mach core to handle the current request. The driver is
unique to the deployed environment. But, it's usually based on an
example from this source tree.
Project State
=============
mach was originally written as a command dispatching framework to aid
Firefox development. While the code is mostly generic, there are still
some pieces that closely tie it to Mozilla/Firefox. The goal is for
these to eventually be removed and replaced with generic features so
mach is suitable for anybody to use. Until then, mach may not be the
best fit for you.
.. toctree::
:maxdepth: 1
commands
driver
logging

100
python/mach/docs/logging.rst Normal file
View file

@ -0,0 +1,100 @@
.. _mach_logging:
=======
Logging
=======
Mach configures a built-in logging facility so commands can easily log
data.
What sets the logging facility apart from most loggers you've seen is
that it encourages structured logging. Instead of conventional logging
where simple strings are logged, the internal logging mechanism logs all
events with the following pieces of information:
* A string *action*
* A dict of log message fields
* A formatting string
Essentially, instead of assembling a human-readable string at
logging-time, you create an object holding all the pieces of data that
will constitute your logged event. For each unique type of logged event,
you assign an *action* name.
Depending on how logging is configured, your logged event could get
written a couple of different ways.
JSON Logging
============
Where machines are the intended target of the logging data, a JSON
logger is configured. The JSON logger assembles an array consisting of
the following elements:
* Decimal wall clock time in seconds since UNIX epoch
* String *action* of message
* Object with structured message data
The JSON-serialized array is written to a configured file handle.
Consumers of this logging stream can just perform a readline() then feed
that into a JSON deserializer to reconstruct the original logged
message. They can key off the *action* element to determine how to
process individual events. There is no need to invent a parser.
Convenient, isn't it?
Logging for Humans
==================
Where humans are the intended consumer of a log message, the structured
log message are converted to more human-friendly form. This is done by
utilizing the *formatting* string provided at log time. The logger
simply calls the *format* method of the formatting string, passing the
dict containing the message's fields.
When *mach* is used in a terminal that supports it, the logging facility
also supports terminal features such as colorization. This is done
automatically in the logging layer - there is no need to control this at
logging time.
In addition, messages intended for humans typically prepends every line
with the time passed since the application started.
Logging HOWTO
=============
Structured logging piggybacks on top of Python's built-in logging
infrastructure provided by the *logging* package. We accomplish this by
taking advantage of *logging.Logger.log()*'s *extra* argument. To this
argument, we pass a dict with the fields *action* and *params*. These
are the string *action* and dict of message fields, respectively. The
formatting string is passed as the *msg* argument, like normal.
If you were logging to a logger directly, you would do something like:
.. code-block:: python
logger.log(logging.INFO, 'My name is {name}',
extra={'action': 'my_name', 'params': {'name': 'Gregory'}})
The JSON logging would produce something like::
[1339985554.306338, "my_name", {"name": "Gregory"}]
Human logging would produce something like::
0.52 My name is Gregory
Since there is a lot of complexity using logger.log directly, it is
recommended to go through a wrapping layer that hides part of the
complexity for you. The easiest way to do this is by utilizing the
LoggingMixin:
.. code-block:: python
import logging
from mach.mixin.logging import LoggingMixin
class MyClass(LoggingMixin):
def foo(self):
self.log(logging.INFO, 'foo_start', {'bar': True},
'Foo performed. Bar: {bar}')

66
python/mach/mach/base.py
View file

@ -2,7 +2,7 @@
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
from __future__ import unicode_literals
from __future__ import absolute_import, unicode_literals
class CommandContext(object):
@ -44,67 +44,3 @@ class UnrecognizedArgumentError(MachError):
self.command = command
self.arguments = arguments
class MethodHandler(object):
"""Describes a Python method that implements a mach command.
Instances of these are produced by mach when it processes classes
defining mach commands.
"""
__slots__ = (
# The Python class providing the command. This is the class type not
# an instance of the class. Mach will instantiate a new instance of
# the class if the command is executed.
'cls',
# Whether the __init__ method of the class should receive a mach
# context instance. This should only affect the mach driver and how
# it instantiates classes.
'pass_context',
# The name of the method providing the command. In other words, this
# is the str name of the attribute on the class type corresponding to
# the name of the function.
'method',
# The name of the command.
'name',
# String category this command belongs to.
'category',
# Description of the purpose of this command.
'description',
# Functions used to 'skip' commands if they don't meet the conditions
# in a given context.
'conditions',
# argparse.ArgumentParser instance to use as the basis for command
# arguments.
'parser',
# Arguments added to this command's parser. This is a 2-tuple of
# positional and named arguments, respectively.
'arguments',
# Argument groups added to this command's parser.
'argument_group_names',
)
def __init__(self, cls, method, name, category=None, description=None,
conditions=None, parser=None, arguments=None,
argument_group_names=None, pass_context=False):
self.cls = cls
self.method = method
self.name = name
self.category = category
self.description = description
self.conditions = conditions or []
self.parser = parser
self.arguments = arguments or []
self.argument_group_names = argument_group_names or []
self.pass_context = pass_context

10
python/mach/mach/commands/commandinfo.py
View file

@ -2,11 +2,12 @@
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, # You can obtain one at http://mozilla.org/MPL/2.0/.
from __future__ import print_function, unicode_literals
from __future__ import absolute_import, print_function, unicode_literals
from mach.decorators import (
CommandProvider,
Command,
CommandArgument,
)
@ -22,11 +23,16 @@ class BuiltinCommands(object):
@Command('mach-debug-commands', category='misc',
description='Show info about available mach commands.')
def debug_commands(self):
@CommandArgument('match', metavar='MATCH', default=None, nargs='?',
help='Only display commands containing given substring.')
def debug_commands(self, match=None):
import inspect
handlers = self.context.commands.command_handlers
for command in sorted(handlers.keys()):
if match and match not in command:
continue
handler = handlers[command]
cls = handler.cls
method = getattr(cls, getattr(handler, 'method'))

7
python/mach/mach/commands/settings.py
View file

@ -2,11 +2,14 @@
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
# You can obtain one at http://mozilla.org/MPL/2.0/.
from __future__ import print_function, unicode_literals
from __future__ import absolute_import, print_function, unicode_literals
from textwrap import TextWrapper
from mach.decorators import Command
from mach.decorators import (
CommandProvider,
Command,
)
#@CommandProvider

2
python/mach/mach/config.py
View file

@ -25,7 +25,7 @@ msgfmt binary to perform this conversion. Generation of the original .po file
can be done via the write_pot() of ConfigSettings.
"""
from __future__ import unicode_literals
from __future__ import absolute_import, unicode_literals
import collections
import gettext

224
python/mach/mach/decorators.py
View file

@ -2,22 +2,97 @@
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
from __future__ import unicode_literals
from __future__ import absolute_import, unicode_literals
import argparse
import collections
import inspect
import types
from .base import (
MachError,
MethodHandler
)
from .base import MachError
from .config import ConfigProvider
from .registrar import Registrar
class _MachCommand(object):
"""Container for mach command metadata.
Mach commands contain lots of attributes. This class exists to capture them
in a sane way so tuples, etc aren't used instead.
"""
__slots__ = (
# Content from decorator arguments to define the command.
'name',
'subcommand',
'category',
'description',
'conditions',
'_parser',
'arguments',
'argument_group_names',
# Describes how dispatch is performed.
# The Python class providing the command. This is the class type not
# an instance of the class. Mach will instantiate a new instance of
# the class if the command is executed.
'cls',
# Whether the __init__ method of the class should receive a mach
# context instance. This should only affect the mach driver and how
# it instantiates classes.
'pass_context',
# The name of the method providing the command. In other words, this
# is the str name of the attribute on the class type corresponding to
# the name of the function.
'method',
# Dict of string to _MachCommand defining sub-commands for this
# command.
'subcommand_handlers',
)
def __init__(self, name=None, subcommand=None, category=None,
description=None, conditions=None, parser=None):
self.name = name
self.subcommand = subcommand
self.category = category
self.description = description
self.conditions = conditions or []
self._parser = parser
self.arguments = []
self.argument_group_names = []
self.cls = None
self.pass_context = None
self.method = None
self.subcommand_handlers = {}
@property
def parser(self):
# Creating CLI parsers at command dispatch time can be expensive. Make
# it possible to lazy load them by using functions.
if callable(self._parser):
self._parser = self._parser()
return self._parser
@property
def docstring(self):
return self.cls.__dict__[self.method].__doc__
def __ior__(self, other):
if not isinstance(other, _MachCommand):
raise ValueError('can only operate on _MachCommand instances')
for a in self.__slots__:
if not getattr(self, a):
setattr(self, a, getattr(other, a))
return self
def CommandProvider(cls):
"""Class decorator to denote that it provides subcommands for Mach.
@ -47,6 +122,8 @@ def CommandProvider(cls):
if len(spec.args) == 2:
pass_context = True
seen_commands = set()
# We scan __dict__ because we only care about the classes own attributes,
# not inherited ones. If we did inherited attributes, we could potentially
# define commands multiple times. We also sort keys so commands defined in
@ -57,45 +134,80 @@ def CommandProvider(cls):
if not isinstance(value, types.FunctionType):
continue
command_name, category, description, conditions, parser = getattr(
value, '_mach_command', (None, None, None, None, None))
if command_name is None:
command = getattr(value, '_mach_command', None)
if not command:
continue
if conditions is None and Registrar.require_conditions:
# Ignore subcommands for now: we handle them later.
if command.subcommand:
continue
seen_commands.add(command.name)
if not command.conditions and Registrar.require_conditions:
continue
msg = 'Mach command \'%s\' implemented incorrectly. ' + \
'Conditions argument must take a list ' + \
'of functions. Found %s instead.'
conditions = conditions or []
if not isinstance(conditions, collections.Iterable):
msg = msg % (command_name, type(conditions))
if not isinstance(command.conditions, collections.Iterable):
msg = msg % (command.name, type(command.conditions))
raise MachError(msg)
for c in conditions:
for c in command.conditions:
if not hasattr(c, '__call__'):
msg = msg % (command_name, type(c))
msg = msg % (command.name, type(c))
raise MachError(msg)
arguments = getattr(value, '_mach_command_args', None)
command.cls = cls
command.method = attr
command.pass_context = pass_context
argument_group_names = getattr(value, '_mach_command_arg_group_names', None)
Registrar.register_command_handler(command)
handler = MethodHandler(cls, attr, command_name, category=category,
description=description, conditions=conditions, parser=parser,
arguments=arguments, argument_group_names=argument_group_names,
pass_context=pass_context)
# Now do another pass to get sub-commands. We do this in two passes so
# we can check the parent command existence without having to hold
# state and reconcile after traversal.
for attr in sorted(cls.__dict__.keys()):
value = cls.__dict__[attr]
Registrar.register_command_handler(handler)
if not isinstance(value, types.FunctionType):
continue
command = getattr(value, '_mach_command', None)
if not command:
continue
# It is a regular command.
if not command.subcommand:
continue
if command.name not in seen_commands:
raise MachError('Command referenced by sub-command does not '
'exist: %s' % command.name)
if command.name not in Registrar.command_handlers:
continue
command.cls = cls
command.method = attr
command.pass_context = pass_context
parent = Registrar.command_handlers[command.name]
if parent._parser:
raise MachError('cannot declare sub commands against a command '
'that has a parser installed: %s' % command)
if command.subcommand in parent.subcommand_handlers:
raise MachError('sub-command already defined: %s' % command.subcommand)
parent.subcommand_handlers[command.subcommand] = command
return cls
class Command(object):
"""Decorator for functions or methods that provide a mach subcommand.
"""Decorator for functions or methods that provide a mach command.
The decorator accepts arguments that define basic attributes of the
command. The following arguments are recognized:
@ -105,8 +217,9 @@ class Command(object):
description -- A brief description of what the command does.
parser -- an optional argparse.ArgumentParser instance to use as
the basis for the command arguments.
parser -- an optional argparse.ArgumentParser instance or callable
that returns an argparse.ArgumentParser instance to use as the
basis for the command arguments.
For example:
@ -114,20 +227,45 @@ class Command(object):
def foo(self):
pass
"""
def __init__(self, name, category=None, description=None, conditions=None,
parser=None):
self._name = name
self._category = category
self._description = description
self._conditions = conditions
self._parser = parser
def __init__(self, name, **kwargs):
self._mach_command = _MachCommand(name=name, **kwargs)
def __call__(self, func):
func._mach_command = (self._name, self._category, self._description,
self._conditions, self._parser)
if not hasattr(func, '_mach_command'):
func._mach_command = _MachCommand()
func._mach_command |= self._mach_command
return func
class SubCommand(object):
"""Decorator for functions or methods that provide a sub-command.
Mach commands can have sub-commands. e.g. ``mach command foo`` or
``mach command bar``. Each sub-command has its own parser and is
effectively its own mach command.
The decorator accepts arguments that define basic attributes of the
sub command:
command -- The string of the command this sub command should be
attached to.
subcommand -- The string name of the sub command to register.
description -- A textual description for this sub command.
"""
def __init__(self, command, subcommand, description=None):
self._mach_command = _MachCommand(name=command, subcommand=subcommand,
description=description)
def __call__(self, func):
if not hasattr(func, '_mach_command'):
func._mach_command = _MachCommand()
func._mach_command |= self._mach_command
return func
class CommandArgument(object):
"""Decorator for additional arguments to mach subcommands.
@ -152,17 +290,16 @@ class CommandArgument(object):
self._command_args = (args, kwargs)
def __call__(self, func):
command_args = getattr(func, '_mach_command_args', [])
if not hasattr(func, '_mach_command'):
func._mach_command = _MachCommand()
command_args.insert(0, self._command_args)
func._mach_command_args = command_args
func._mach_command.arguments.insert(0, self._command_args)
return func
class CommandArgumentGroup(object):
"""Decorator for additional argument groups to mach subcommands.
"""Decorator for additional argument groups to mach commands.
This decorator should be used to add arguments groups to mach commands.
Arguments to the decorator are proxied to
@ -185,11 +322,10 @@ class CommandArgumentGroup(object):
self._group_name = group_name
def __call__(self, func):
command_arg_group_names = getattr(func, '_mach_command_arg_group_names', [])
if not hasattr(func, '_mach_command'):
func._mach_command = _MachCommand()
command_arg_group_names.insert(0, self._group_name)
func._mach_command_arg_group_names = command_arg_group_names
func._mach_command.argument_group_names.insert(0, self._group_name)
return func

169
python/mach/mach/dispatcher.py
View file

@ -2,7 +2,7 @@
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
from __future__ import unicode_literals
from __future__ import absolute_import, unicode_literals
import argparse
import difflib
@ -11,6 +11,7 @@ import sys
from operator import itemgetter
from .base import (
MachError,
NoCommandError,
UnknownCommandError,
UnrecognizedArgumentError,
@ -95,37 +96,71 @@ class CommandAction(argparse.Action):
if command == 'help':
if args and args[0] not in ['-h', '--help']:
# Make sure args[0] is indeed a command.
self._handle_subcommand_help(parser, args[0])
self._handle_command_help(parser, args[0])
else:
self._handle_main_help(parser, namespace.verbose)
sys.exit(0)
elif '-h' in args or '--help' in args:
# -h or --help is in the command arguments.
self._handle_subcommand_help(parser, command)
self._handle_command_help(parser, command)
sys.exit(0)
else:
raise NoCommandError()
# Command suggestion
if command not in self._mach_registrar.command_handlers:
# Make sure we don't suggest any deprecated commands.
names = [h.name for h in self._mach_registrar.command_handlers.values()
if h.cls.__name__ == 'DeprecatedCommands']
# We first try to look for a valid command that is very similar to the given command.
suggested_commands = difflib.get_close_matches(command, self._mach_registrar.command_handlers.keys(), cutoff=0.8)
suggested_commands = difflib.get_close_matches(command, names, cutoff=0.8)
# If we find more than one matching command, or no command at all, we give command suggestions instead
# (with a lower matching threshold). All commands that start with the given command (for instance: 'mochitest-plain',
# 'mochitest-chrome', etc. for 'mochitest-') are also included.
if len(suggested_commands) != 1:
suggested_commands = set(difflib.get_close_matches(command, self._mach_registrar.command_handlers.keys(), cutoff=0.5))
suggested_commands |= {cmd for cmd in self._mach_registrar.command_handlers if cmd.startswith(command)}
suggested_commands = set(difflib.get_close_matches(command, names, cutoff=0.5))
suggested_commands |= {cmd for cmd in names if cmd.startswith(command)}
raise UnknownCommandError(command, 'run', suggested_commands)
sys.stderr.write("We're assuming the '%s' command is '%s' and we're executing it for you.\n\n" % (command, suggested_commands[0]))
command = suggested_commands[0]
handler = self._mach_registrar.command_handlers.get(command)
# FUTURE
# If we wanted to conditionally enable commands based on whether
# it's possible to run them given the current state of system, here
# would be a good place to hook that up.
usage = '%(prog)s [global arguments] ' + command + \
' [command arguments]'
subcommand = None
# If there are sub-commands, parse the intent out immediately.
if handler.subcommand_handlers:
if not args:
self._handle_subcommand_main_help(parser, handler)
sys.exit(0)
elif len(args) == 1 and args[0] in ('help', '--help'):
self._handle_subcommand_main_help(parser, handler)
sys.exit(0)
# mach <command> help <subcommand>
elif len(args) == 2 and args[0] == 'help':
subcommand = args[1]
subhandler = handler.subcommand_handlers[subcommand]
self._handle_subcommand_help(parser, command, subcommand, subhandler)
sys.exit(0)
# We are running a sub command.
else:
subcommand = args[0]
if subcommand[0] == '-':
raise MachError('%s invoked improperly. A sub-command name '
'must be the first argument after the command name.' %
command)
if subcommand not in handler.subcommand_handlers:
raise UnknownCommandError(subcommand, 'run',
handler.subcommand_handlers.keys())
handler = handler.subcommand_handlers[subcommand]
usage = '%(prog)s [global arguments] ' + command + ' ' + \
subcommand + ' [command arguments]'
args.pop(0)
# We create a new parser, populate it with the command's arguments,
# then feed all remaining arguments to it, merging the results
@ -134,12 +169,12 @@ class CommandAction(argparse.Action):
parser_args = {
'add_help': False,
'usage': '%(prog)s [global arguments] ' + command +
' [command arguments]',
'usage': usage,
}
if handler.parser:
subparser = handler.parser
subparser.context = self._context
else:
subparser = argparse.ArgumentParser(**parser_args)
@ -166,6 +201,7 @@ class CommandAction(argparse.Action):
# not interfere with arguments passed to the command.
setattr(namespace, 'mach_handler', handler)
setattr(namespace, 'command', command)
setattr(namespace, 'subcommand', subcommand)
command_namespace, extra = subparser.parse_known_args(args)
setattr(namespace, 'command_args', command_namespace)
@ -251,12 +287,31 @@ class CommandAction(argparse.Action):
parser.print_help()
def _handle_subcommand_help(self, parser, command):
def _populate_command_group(self, parser, handler, group):
extra_groups = {}
for group_name in handler.argument_group_names:
group_full_name = 'Command Arguments for ' + group_name
extra_groups[group_name] = \
parser.add_argument_group(group_full_name)
for arg in handler.arguments:
# Apply our group keyword.
group_name = arg[1].get('group')
if group_name:
del arg[1]['group']
group = extra_groups[group_name]
group.add_argument(*arg[0], **arg[1])
def _handle_command_help(self, parser, command):
handler = self._mach_registrar.command_handlers.get(command)
if not handler:
raise UnknownCommandError(command, 'query')
if handler.subcommand_handlers:
self._handle_subcommand_main_help(parser, handler)
return
# This code is worth explaining. Because we are doing funky things with
# argument registration to allow the same option in both global and
# command arguments, we can't simply put all arguments on the same
@ -274,6 +329,7 @@ class CommandAction(argparse.Action):
if handler.parser:
c_parser = handler.parser
c_parser.context = self._context
c_parser.formatter_class = NoUsageFormatter
# Accessing _action_groups is a bit shady. We are highly dependent
# on the argparse implementation not changing. We fail fast to
@ -293,31 +349,84 @@ class CommandAction(argparse.Action):
c_parser = argparse.ArgumentParser(**parser_args)
group = c_parser.add_argument_group('Command Arguments')
extra_groups = {}
for group_name in handler.argument_group_names:
group_full_name = 'Command Arguments for ' + group_name
extra_groups[group_name] = \
c_parser.add_argument_group(group_full_name)
self._populate_command_group(c_parser, handler, group)
for arg in handler.arguments:
# Apply our group keyword.
group_name = arg[1].get('group')
if group_name:
del arg[1]['group']
group = extra_groups[group_name]
group.add_argument(*arg[0], **arg[1])
# This will print the description of the command below the usage.
description = handler.description
if description:
parser.description = description
# Set the long help of the command to the docstring (if present) or
# the command decorator description argument (if present).
if handler.docstring:
parser.description = format_docstring(handler.docstring)
elif handler.description:
parser.description = handler.description
parser.usage = '%(prog)s [global arguments] ' + command + \
' [command arguments]'
# This is needed to preserve line endings in the description field,
# which may be populated from a docstring.
parser.formatter_class = argparse.RawDescriptionHelpFormatter
parser.print_help()
print('')
c_parser.print_help()
def _handle_subcommand_main_help(self, parser, handler):
parser.usage = '%(prog)s [global arguments] ' + handler.name + \
' subcommand [subcommand arguments]'
group = parser.add_argument_group('Sub Commands')
for subcommand, subhandler in sorted(handler.subcommand_handlers.iteritems()):
group.add_argument(subcommand, help=subhandler.description,
action='store_true')
if handler.docstring:
parser.description = format_docstring(handler.docstring)
parser.formatter_class = argparse.RawDescriptionHelpFormatter
parser.print_help()
def _handle_subcommand_help(self, parser, command, subcommand, handler):
parser.usage = '%(prog)s [global arguments] ' + command + \
' ' + subcommand + ' [command arguments]'
c_parser = argparse.ArgumentParser(add_help=False,
formatter_class=CommandFormatter)
group = c_parser.add_argument_group('Sub Command Arguments')
self._populate_command_group(c_parser, handler, group)
if handler.docstring:
parser.description = format_docstring(handler.docstring)
parser.formatter_class = argparse.RawDescriptionHelpFormatter
parser.print_help()
print('')
c_parser.print_help()
class NoUsageFormatter(argparse.HelpFormatter):
def _format_usage(self, *args, **kwargs):
return ""
def format_docstring(docstring):
"""Format a raw docstring into something suitable for presentation.
This function is based on the example function in PEP-0257.
"""
if not docstring:
return ''
lines = docstring.expandtabs().splitlines()
indent = sys.maxint
for line in lines[1:]:
stripped = line.lstrip()
if stripped:
indent = min(indent, len(line) - len(stripped))
trimmed = [lines[0].strip()]
if indent < sys.maxint:
for line in lines[1:]:
trimmed.append(line[indent:].rstrip())
while trimmed and not trimmed[-1]:
trimmed.pop()
while trimmed and not trimmed[0]:
trimmed.pop(0)
return '\n'.join(trimmed)

59
python/mach/mach/main.py
View file

@ -25,7 +25,11 @@ from .base import (
UnrecognizedArgumentError,
)
from .decorators import CommandProvider
from .decorators import (
CommandArgument,
CommandProvider,
Command,
)
from .config import ConfigSettings
from .dispatcher import CommandAction
@ -87,13 +91,6 @@ It looks like you passed an unrecognized argument into mach.
The %s command does not accept the arguments: %s
'''.lstrip()
INVALID_COMMAND_CONTEXT = r'''
It looks like you tried to run a mach command from an invalid context. The %s
command failed to meet the following conditions: %s
Run |mach help| to show a list of all commands available to the current context.
'''.lstrip()
INVALID_ENTRY_POINT = r'''
Entry points should return a list of command providers or directories
containing command providers. The following entry point is invalid:
@ -153,7 +150,7 @@ class ContextWrapper(object):
except AttributeError as e:
try:
ret = object.__getattribute__(self, '_handler')(self, key)
except AttributeError, TypeError:
except (AttributeError, TypeError):
# TypeError is in case the handler comes from old code not
# taking a key argument.
raise e
@ -421,41 +418,17 @@ To see more help for a specific command, run:
raise MachError('ArgumentParser result missing mach handler info.')
handler = getattr(args, 'mach_handler')
cls = handler.cls
if handler.pass_context:
instance = cls(context)
else:
instance = cls()
if handler.conditions:
fail_conditions = []
for c in handler.conditions:
if not c(instance):
fail_conditions.append(c)
if fail_conditions:
print(self._condition_failed_message(handler.name, fail_conditions))
return 1
fn = getattr(instance, handler.method)
try:
result = fn(**vars(args.command_args))
if not result:
result = 0
assert isinstance(result, (int, long))
return result
return Registrar._run_command_handler(handler, context=context,
debug_command=args.debug_command, **vars(args.command_args))
except KeyboardInterrupt as ki:
raise ki
except Exception as e:
exc_type, exc_value, exc_tb = sys.exc_info()
# The first frame is us and is never used.
stack = traceback.extract_tb(exc_tb)[1:]
# The first two frames are us and are never used.
stack = traceback.extract_tb(exc_tb)[2:]
# If we have nothing on the stack, the exception was raised as part
# of calling the @Command method itself. This likely means a
@ -502,16 +475,6 @@ To see more help for a specific command, run:
self.logger.log(level, format_str,
extra={'action': action, 'params': params})
@classmethod
def _condition_failed_message(cls, name, conditions):
msg = ['\n']
for c in conditions:
part = [' %s' % c.__name__]
if c.__doc__ is not None:
part.append(c.__doc__)
msg.append(' - '.join(part))
return INVALID_COMMAND_CONTEXT % (name, '\n'.join(msg))
def _print_error_header(self, argv, fh):
fh.write('Error running mach:\n\n')
fh.write(' ')
@ -598,6 +561,8 @@ To see more help for a specific command, run:
global_group.add_argument('-h', '--help', dest='help',
action='store_true', default=False,
help='Show this help message.')
global_group.add_argument('--debug-command', action='store_true',
help='Start a Python debugger when command is dispatched.')
for args, kwargs in self.global_arguments:
global_group.add_argument(*args, **kwargs)

69
python/mach/mach/registrar.py
View file

@ -2,10 +2,17 @@
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
from __future__ import unicode_literals
from __future__ import absolute_import, unicode_literals
from .base import MachError
INVALID_COMMAND_CONTEXT = r'''
It looks like you tried to run a mach command from an invalid context. The %s
command failed to meet the following conditions: %s
Run |mach help| to show a list of all commands available to the current context.
'''.lstrip()
class MachRegistrar(object):
"""Container for mach command and config providers."""
@ -38,15 +45,17 @@ class MachRegistrar(object):
self.categories[name] = (title, description, priority)
self.commands_by_category[name] = set()
def dispatch(self, name, context=None, **args):
"""Dispatch/run a command.
@classmethod
def _condition_failed_message(cls, name, conditions):
msg = ['\n']
for c in conditions:
part = [' %s' % c.__name__]
if c.__doc__ is not None:
part.append(c.__doc__)
msg.append(' - '.join(part))
return INVALID_COMMAND_CONTEXT % (name, '\n'.join(msg))
Commands can use this to call other commands.
"""
# TODO The logic in this function overlaps with code in
# mach.main.Main._run() and should be consolidated.
handler = self.command_handlers[name]
def _run_command_handler(self, handler, context=None, debug_command=False, **kwargs):
cls = handler.cls
if handler.pass_context and not context:
@ -57,9 +66,49 @@ class MachRegistrar(object):
else:
instance = cls()
if handler.conditions:
fail_conditions = []
for c in handler.conditions:
if not c(instance):
fail_conditions.append(c)
if fail_conditions:
print(self._condition_failed_message(handler.name, fail_conditions))
return 1
fn = getattr(instance, handler.method)
return fn(**args) or 0
if debug_command:
import pdb
result = pdb.runcall(fn, **kwargs)
else:
result = fn(**kwargs)
result = result or 0
assert isinstance(result, (int, long))
return result
def dispatch(self, name, context=None, argv=None, **kwargs):
"""Dispatch/run a command.
Commands can use this to call other commands.
"""
# TODO handler.subcommand_handlers are ignored
handler = self.command_handlers[name]
if handler.parser:
parser = handler.parser
# save and restore existing defaults so **kwargs don't persist across
# subsequent invocations of Registrar.dispatch()
old_defaults = parser._defaults.copy()
parser.set_defaults(**kwargs)
kwargs, _ = parser.parse_known_args(argv or [])
kwargs = vars(kwargs)
parser._defaults = old_defaults
return self._run_command_handler(handler, context=context, **kwargs)
Registrar = MachRegistrar()

2
python/mach/mach/terminal.py
View file

@ -8,7 +8,7 @@ All the terminal interaction code is consolidated so the complexity can be in
one place, away from code that is commonly looked at.
"""
from __future__ import print_function, unicode_literals
from __future__ import absolute_import, print_function, unicode_literals
import logging
import sys

1
python/mach/mach/test/common.py
View file

@ -9,6 +9,7 @@ import os
import unittest
from mach.main import Mach
from mach.base import CommandContext
here = os.path.abspath(os.path.dirname(__file__))

2
python/mach/mach/test/providers/throw.py
View file

@ -4,6 +4,8 @@
from __future__ import unicode_literals
import time
from mach.decorators import (
CommandArgument,
CommandProvider,

5
python/mach/mach/test/test_conditions.py
View file

@ -8,6 +8,7 @@ import os
from mach.base import MachError
from mach.main import Mach
from mach.registrar import Registrar
from mach.test.common import TestBase
from mozunit import main
@ -48,14 +49,14 @@ class TestConditions(TestBase):
result, stdout, stderr = self._run_mach([name])
self.assertEquals(1, result)
fail_msg = Mach._condition_failed_message(name, fail_conditions)
fail_msg = Registrar._condition_failed_message(name, fail_conditions)
self.assertEquals(fail_msg.rstrip(), stdout.rstrip())
for name in ('cmd_bar_ctx', 'cmd_foobar_ctx'):
result, stdout, stderr = self._run_mach([name], _populate_context)
self.assertEquals(1, result)
fail_msg = Mach._condition_failed_message(name, fail_conditions)
fail_msg = Registrar._condition_failed_message(name, fail_conditions)
self.assertEquals(fail_msg.rstrip(), stdout.rstrip())
def test_invalid_type(self):

2
python/mach/mach/test/test_entry_point.py
View file

@ -11,6 +11,8 @@ from mach.base import MachError
from mach.test.common import TestBase
from mock import patch
from mozunit import main
here = os.path.abspath(os.path.dirname(__file__))