Xi-CAM Documentation

Xi-CAM Documentation

The Xi-CAM package is a plugin-based framework and graphical user interface (GUI) application for synchrotron data management, acquisition, visualization, and analysis. Support for a variety of techniques is provided through its plugin architecture.

Getting Started

Before starting to develop plugins, you will need to install Xi-CAM and some of its dependencies.

Follow the QuickStart guide to get started with installation and exploring an example Xi-CAM plugin (covers GUIPlugin, Workflow, and OperationPlugins).

For more detailed documentation, you may want to visit the following:

The Resources section has useful links to tutorials, examples, and documentation that can help with developing Xi-CAM plugins.

Developer Documentation

QuickStart Guide

This is a quick-start guide that will help you install Xi-CAM and explore an example plugin that you can experiment with.

This guide does not explore the implementation of the plugin in too much detail. For more in-depth documentation for developing plugins from scratch, see:

GUIPlugin Documentation

This documentation provides information on GUIPlugins and GUILayouts to help with designing your own plugins for Xi-CAM. API reference documentation is also included at the bottom.

If you are new to developing Xi-CAM plugins, it is recommended that you follow the quick-start documentation first.

For more general development resources, see the resources documentation.

What Is A GUIPlugin?

A GUIPlugin is an interactive user-facing plugin in Xi-CAM. It can be used to visualize and analyze data.

GUIPlugins make use of the qtpy Python package for interactive GUI components. See the resources documentation for more information.

Where is GUIPlugin?
xicam.plugins.guiplugin
What Does a GUIPlugin Look Like?

First, let’s look at what Xi-CAM looks like when you first load it:

Main window of Xi-CAM on startup, with three installed plugins.

Main window of Xi-CAM when running xicam. Note that there are three installed GUIPlugins here; if you haven’t installed any plugins, you won’t see any listed.

As you can see, the main window of Xi-CAM after it has finished loading shows any installed GUIPlugins, a citation / references widget, a preview widget, and a data browser widget. The data browser widget can be used to load data into a GUIPlugin. The data preview widget can be used to “preview” data before loading it.

It is important to keep in mind a few concepts for GUIPlugins:

  • A GUIPlugin can have one or more stages.

  • Each stage is defined with a GUILayout.

  • A GUILayout is defined with a widget (or multiple widgets).

These concepts are explored in more detail later in this document.

How Do I Create a GUIPlugin?

To create a GUIPlugin, you will need:

  • a derived class of GUIPlugin

  • a setup.py file with a xicam.plugins.GUIPlugin entry point

Although you may structure your plugin’s code and support files as you like, we recommend using a cookiecutter template that we have created for Xi-CAM’s GUIPlugin.

What is cookiecutter?

cookiecutter is a templating tool that can be used to interactively create python project. For more information, see the cookiecutter documentation.

Install cookiecutter

In your active environment, you will need to pip install cookiecutter.

Run cookiecutter with the Xi-CAM GUIPlugin Template

Now, in the directory of your choice (the home directory, ~, should work if you are unsure), run the following:

cookiecutter https://github.com/Xi-CAM/Xi-cam.templates.GuiPlugin

This will download the template for creating a GUIPlugin, then present you with a series of prompts.

A prompt will look like prompt [default value]: . If you want to use the default value specified, hit the enter key. Otherwise, respond to the prompt with the value you would like.

Here are the prompts with their descriptions:

prompt default value description
package_name my_plugin name of the package to create (will be xicam.package_name)
plugin_name MyPlugin name of the GUIPlugin class to create (name of your gui plugin in code)
display_name My Plugin name of the GUIPlugin as it will appear in the Xi-CAM GUI
plugin_version 1.0.0 current plugin version number
author_name author's name
author_email author's email
author_url url for the author/plugin (e.g. plugin GitHub repo url)
description description of the plugin
dependencies packages the plugin depends on
keywords [Xi-cam, plugin] keywords to tag the plugin with (when packaging)

This will create a python package with some files and code to get started developing a GUIPlugin. You can always change the names of your plugin, package, etc. later by hand.

The GUIPlugin you created will be implemented in xicam.package_name/xicam/package_name/__init__.py.

For purposes of this documentation, we will refer to these values by their defaults.

Installing Your GUIPlugin

When you create a new plugin package using cookiecutter, one of the files it generates is setup.py. This contains meta-information about the package. When you run pip install of your package, it uses this information to create a disribution.

setup.py also defines entry points, which Xi-CAM uses to find plugins.

For more information about entry points in Xi-CAM, see the following documentation.

Navigate to your created package directory and create an editable pip install:

cd xicam.my_plugin
pip install -e .

This tells pip install your file locally by looking at the setup.py file, and the -e allows you to make changes to your code without having to reinstall.

If you change an entry point in setup.py, you must reinstall.

Selecting and Activating a GUIPlugin

We can activate any of the installed GUIPlugins by clicking on their name at the top. Let’s click on “My Plugin”:

Xi-CAM MyPlugin

MyPlugin’s interface.

Note that this plugin doesn’t do much yet; it simply displays the text “Stage 1…” You can also click “Stage 2” at the top, and you will see the text “Stage 2…” in the center.

How is MyPlugin Implemented?

The code for MyPlugin is implemented in xicam.package_name/xicam/package_name/__init__.py.

from qtpy.QtWidgets import QLabel

from xicam.plugins import GUIPlugin, GUILayout


class MyPlugin(GUIPlugin):
    # Defines the name of the plugin (how it is displayed in Xi-CAM)
    name = "My Plugin"

    def __init__(self, *args, **kwargs):
        # Insert code here

        # Modify stages here
        self.stages = {'Stage 1': GUILayout(QLabel("Stage 1...")),
                       "Stage 2": GUILayout(QLabel("Stage 2..."))}

        # Initialize the parent class, GUIPlugin
        super(MyPlugin, self).__init__(*args, **kwargs)

Cookiecutter set up this starter code for us. We have a derived version of GUIPlugin, which we call MyPlugin. It has the name “My Plugin”, which is how it will appear in the Xi-CAM GUI.

We then have an __init__ method to describe how to create a MyPlugin. Notice that there is a QLabel, which is simply text, added into two GUILayouts. These layouts are then added to the interface via self.stages.

What Is a Stage?

Visually, a stage is a stand-alone interface for a GUIPlugin. A GUIPlugin must have at least one stage but may have multiple stages. With multiple stages, each stage has its own interface and each stage can be selected in the top bar of Xi-CAM.

Stages for a GUIPlugin are accessible with self.stages. self.stages is a dictionary where each

  • key is the name of the stage

  • value is a GUILayouts

For example, we might define two stages as:

self.stages = {"A": GUILayout(QLabel("1")),
               "B": GUILayout(QLabel("2"))}

This will look like:

MyPlugin with multiple stages

The interface of a plugin named “My Plugin” with multiple stages, “A” and “B”. Note that “A” is currently selected, so we see the label “1” in the middle of the window.

What Is a GUILayout?

A GUILayout is a layout used to describe how widgets should be organized in a stage in a GUIPlugin.

Layout of Xi-CAM, corresponding to a GUILayout.

The layout corresponds to a 3x3 grid in the Xi-CAM main window, with the names center, left, right, lefttop, righttop, leftbottom, rightbottom. These names correspond to the arguments you can pass when creating a GUILayout.

You must provide at least one widget, which will be the center widget.

What Is a QLabel?

QLabel is a Qt widget provided by the Qt backend Xi-CAM makes use of. It acts a widget that holds simple text.

For more information on Qt, see Qt for Python Documentation.

How Do I Load Data into My Plugin?

In order to load data into a GUIPlugin, you must:

  • have access to or configure a databroker catalog

  • re-implement appendCatalog in your GUIPlugin derived class

    • this will need to have access to an internal widget to display the data

  • have a GUIPlugin selected in Xi-CAM

Configuring a Databroker Catalog

For purposes of this documentation, we will be using a sample msgpack catalog and a starter catalog.yml file you can download.

For general help about databroker, catalogs, and configuration, there is excellent documentation here: https://nsls-ii.github.io/databroker/v2/index.html. Additional documentation about catalogs can be found here: https://intake.readthedocs.io/en/latest/index.html

Download

MD5

349497da-ead2-4015-8201-4719f8a2de69.msgpack

3a18341f570b100afbaff1c889e9b4f8

catalog.yml

c14814b4537810f14300f8c8d5949285

After downloading these files, we will want to do three things:

  1. Decide where to put our data and move it there

  2. Update our catalog.yml paths to have a path directory the data is in

  3. Move our catalog.yml to a place it can be discovered

Moving the msgpack Data

You can choose where you’d like to copy or move your data. For purposes of this guide, we will create a new directory in our home called catalogs and move the downloaded msgpack file there.

Updating catalog.yml

Now that we’ve moved / copied our sample catalog msgpack file, we need to update our catalog.yml to tell it where it can find that data.

We will want to add a line under paths in catalog.yml that is the complete file path to the catalogs directory we added above.

Making catalog.yml Discoverable

To know where we can put our catalog.yml file, we can run the following in a Python interpreter:

from databroker import catalog_search_path
print(catalog_search_path())

You can move the catalog.yml file in any of the paths listed. Note that typically there will be a more user-oriented path and a more global system-level path for the catalogs to find. You can copy the catalog.yml file to either (or both) path depending on how you want a machine set up.

Implementing appendCatalog

Let’s implement the appendCatalog method in MyPlugin so we can load the catalog. We will also be adding a widget to view the loaded catalog.

Inside of the MyPlugin class (located in xicam/my_plugin/__init__.py), add the appendCatalog as follows:

from qtpy.QtWidgets import QLabel

from xicam.core.msg import logMessage
from xicam.plugins import GUILayout, GUIPlugin
from xicam.gui.widgets.imageviewmixins import CatalogView


class MyPlugin(GUIPlugin):
    # Define the name of the plugin (how it is displayed in Xi-CAM)
    name = "My Plugin"

    def __init__(self, *args, **kwargs):
        self._catalog_viewer = CatalogView()
        self._stream = "primary"
        self._field = "img"

        catalog_viewer_layout = GUILayout(self._catalog_viewer)
                
        # Modify stages here
        # self.stages = {"Stage 1": GUILayout(QLabel("Stage 1..."))}
        self.stages = {"View Catalog": catalog_viewer_layout}

        super(MyPlugin, self).__init__(*args, **kwargs)

    def appendCatalog(self, catalog):
        self._catalog_viewer.setCatalog(catalog, self._stream, self._field)
        logMessage(f"Opening catalog with stream {self._stream} and field {self._field}.")
API Reference
class xicam.plugins.guiplugin.GUIPlugin

GUIPlugin class for interactive Xi-CAM plugins.

This class represents the fundamental interactive plugin for Xi-CAM.

GUIPlugins are left uninstanciated until all plugins are loaded so that all dependent widgets are loaded before the UI is setup. They DO become singletons.

appendCatalog(catalog: BlueskyRun, **kwargs)

Re-implement to define how to add a catalog to your GUIPlugin.

Parameters

catalog (BlueskyRun) – Catalog reference that you can use as you wish (corresponds to the opened catalog in DataResourceBrowser).

property stages: OrderedDict

Returns the stages of the GUIPlugin.

A stage is defined by a GUILayout; each stage represents a distinct user-interface in a GUIPlugin.

class xicam.plugins.guiplugin.GUILayout(center, left=PanelState.Defaulted, right=PanelState.Defaulted, bottom=PanelState.Defaulted, top=PanelState.Defaulted, lefttop=PanelState.Defaulted, righttop=PanelState.Defaulted, leftbottom=PanelState.Defaulted, rightbottom=PanelState.Defaulted)

Represents a layout of dockable widgets in a 3x3 grid.

The parameters can either be a PanelState value or a QWidget object. Note that only the center parameter is required; the other parameters default to PanelState.Defaulted. The default behavior of a PanelState.Defaulted widget is to be hidden.

Parameters

  • center (Union[QWidget, PanelState]) – The center widget

  • left (Union[QWidget, PanelState], optional) – The left widget

  • right (Union[QWidget, PanelState], optional) – The right widget

  • bottom (Union[QWidget, PanelState], optional) – The bottom widget

  • top (Union[QWidget, PanelState], optional) – The top widget

  • lefttop (Union[QWidget, PanelState], optional) – The top-left widget

  • righttop (Union[QWidget, PanelState], optional) – The top-right widget

  • leftbottom (Union[QWidget, PanelState], optional) – The bottom-left widget

  • rightbottom (Union[QWidget, PanelState], optional) – The bottom-right widget

Notes

For an example of how this class can be used, see the xicam.gui XicamMainWindow class.

OperationPlugin Documentation

This documentation provides information on the foundational aspects of the OperationPlugin class, as well as a more detailed API reference.

If you are new to developing Xi-CAM plugins, it is recommended that you follow the quick-start documentation first.

For more general development resources, see the Resources page.

What Is an OperationPlugin?

An OperationPlugin can be thought of as a function with some extra annotations attached to it. When we want to define an OperationPlugin, we simply need to define a Python function, then add some additional syntax to the function to define things like inputs, outputs, descriptions of inputs/outputs, units, etc.

To achieve this, The OperationClass makes extensive use of Python decorators.

Where Is OperationPlugin?
xicam.plugins.operationplugin
What Does an OperationPlugin Look Like?

Let’s start off with a simple function that computes the square of its input:

def my_square(n):
    return n**2

Now, let’s make this an OperationPlugin:

from xicam.plugins.operationplugin import operation, output_names

@operation
@output_names("square")
def my_square(n):
    return n**2

That’s it!

Notice the two decorators here: @operation and @output_names.

The @operation says that this function is now a Xi-CAM OperationPlugin. Any input arguments for the function will be the input names for the operation. In this case, our input is n. (This can actually be overwritten by using a different decorator, @input_names, which is described later.)

The @output_names allows us to name our outputs, in this case, square. This will be useful when connecting multiple operations together in a Workflow.

Default Input Values

If you want to provide your operation with default input values, you can use argument defaults in your function:

from xicam.plugins.operationplugin import operation, output_names

@operation
@output_names("square")
def my_square(n = 0):
    return n**2

This provides this operation’s n input with a default value of 0.

Required and Highly-Used Decorators

In order to make a function an operation, the following decorators must be used:

  • @operation – allows creation of operations from the function

  • @output_names – defines the name of the output(s)

Additionally, although not required to for an operation, the following decorators are highly-recommended for use:

  • @display_name – the name of the operation

  • @describe_input – attach a description to the specified input (can be used multiple times)

  • @describe_output – attach a description to the specified output (can be used multiple times)

Type Hinting (Optional)

With Python3 (3.5+), you can add type hinting to your code. In the context of Xi-CAM OperationPlugins, this can be used to make your operation code a little easier to read.

Let’s use the my_square function we defined earlier in this operation:

from xicam.plugins.operationplugin import operation, output_names

@operation
@output_names("square")
def my_square(n: int) -> int:
    return n**2

Note the n: int and the -> int: here. These suggest (but do not mandate) that the input be an integer, and the output expected is an integer.

Again, these are not required, but they can help with readability and debugging your code.

For more information, see Python’s typing module.

Example

A simple division operation that returns both the quotient and remainder.

This illustrates the use of multiple input/output descriptions and multiple outputs.

from typing import Tuple
from xicam.plugins.operationplugin import describe_input, describe_output, display_name, operation, output_names

@operation
@output_names("quotient", "remainder")
@display_name("Division with Remainder")
@describe_input("dividend", "The number being divided.")
@describe_input("divisor", "The number to divide by.")
@describe_output("quotient", "The result of the division.")
@describe_output("remainder", "The remaining value.")
def my_divide(dividend: int, divisor: int = 1) -> Tuple[int, int]:
    quotient = int(dividend // divisor)
    remainder = dividend % divisor
    return quotient, remainder
How Do I Use an OperationPlugin?

Now that we’ve defined an operation, how do we actually use it?

When we define an operation using the @operation decorator around a function, we are defining a new operation class.

We can then create an operation object by using the syntax func(), where func is the name of the function in the operation.

Let’s take our my_square operation (defined above) and create one:

from xicam.plugins.operationplugin import operation, output_names

@operation
@output_names("square")
def my_square(n):
    return n**2

op = my_square()

Now that we have an operation object (instance), op, we can use it within a Workflow.

Let’s create a Workflow, add our operation to it, then execute it.

from xicam.core.execution import Workflow
from xicam.plugins.operationplugin import operation, output_names

@operation
@output_names("square")
def my_square(n):
    return n**2

op = my_square()
workflow = Workflow()
workflow.add_operation(op)
result = workflow.execute(n=11).result()
print(result)

We create a my_square operation, create a Workflow, and add the operation to the Workflow. Then, we execute the Workflow, sending in the input n=11, wait for the result, and print it.

(For purposes of this document, we won’t cover Workflow in depth. More information about Workflow can be found in the Workflow Documentation.)

API Documentation
@xicam.plugins.operationplugin.operation(func: Callable, filled_values: Optional[dict] = None, fixable: Optional[dict] = None, fixed: Optional[dict] = None, input_names: Optional[Tuple[str, ...]] = None, output_names: Optional[Tuple[str, ...]] = None, limits: Optional[dict] = None, opts: Optional[dict] = None, output_shape: Optional[dict] = None, units: Optional[dict] = None, visible: Optional[dict] = None, name: Optional[str] = None, input_descriptions: Optional[dict] = None, output_descriptions: Optional[dict] = None, categories: Optional[Sequence[Union[tuple, str]]] = None) → Type[OperationPlugin]

Create a new operation.

When you define a new operation, you must use this decorator (`@operation`) and the `@output_names` decorator.

This function can be used as a decorator to define a new operation type. The operation can then be instantiated by using the () operator on the operation function’s name.

Parameters

  • func (Callable) – Function that this operation will call.

  • filled_values (dict, optional) – Values to fill for the parameters.

  • fixable (dict, optional) – Indicates which parameters are able to be fixed.

  • fixed (dict, optional) – Indicates whether or not a parameter is fixed.

  • limits (dict, optional) – Defines limits for parameters.

  • opts (dict, optional) – Additional options (kwargs) for the parameter (useful with pyqtgraph’s Parameter/ParameterTree).

  • output_names (tuple, optional) – Names for the outputs, or returned values, of the operation.

  • output_shape (dict, optional) – Defines expected shapes for the outputs.

  • units (dict, optional) – Defines units for the parameters in the operation.

  • name (str, optional) – The display name to be shown to the user. Defaults to self.__name__

  • visible (dict, optional) – Indicates if a parameter is visible or not (see pyqtgraph.Parameter).

  • input_descriptions (dict, optional) – A mapping dict containing descriptions for each named input

  • output_descriptions (dict, optional) – A mapping dict containing descriptions for each named output

  • categories (List[Union[tuple, str], optional) – A sequence of categories to associate with this operation.

Example

Create a new operation type and create a new operation instance from it.

>>> from xicam.core.execution import Workflow
>>> from xicam.plugins.operationplugin import operation, output_names
>>> @operation
>>> @output_names("my_output")
>>> def my_func(x: float = 0.0) -> float:
>>>     return x * -1
>>> op = my_func()
>>> workflow = Workflow()
>>> result = workflow.execute(x=2.5).result()
>>> print(result)
@xicam.plugins.operationplugin.output_names

Decorator to define the names of the outputs for an operation.

Defines N-number of output names. These names will be used (in-order) to define any outputs that the operation has.

Parameters

names (List[str]) – Names for the outputs in the operation.

Example

Define an operation that has the outputs x and y.

>>> @OperationPlugin
>>> @output_names("x", "y")
>>> def some_operation(a: int, b: int) -> Tuple[int, int]:
>>>     return a, b
@xicam.plugins.operationplugin.categories(*categories: Union[tuple, str])

Decorator to assign categories to a operation.

These categories will be used to populate the structure of Xi-cam’s menus of OperationPlugins.

Parameters

categories (Tuple[Union[tuple, str]]) – A sequence of categories. Each item is a tuple or str. If an item is a tuple, each item in the tuple is considered as an additional depth in the menu structure.

Example

Define an operation that is in the following categories:

Generic Functions
    Simple Math
        (square operation)
Math Functions
        (square operation)

>>> @OperationPlugin
>>> @categories(('Generic Functions', 'Simple Math'), 'Math Functions')
>>> def square(x: int = 100) -> int:
>>>     return x**2
@xicam.plugins.operationplugin.describe_input(arg_name: str, description: str)

Decorator to set the description for input arg_name.

This is useful for annotating the parameter with additional information for users.

These annotations are displayed in GUI representations of the operation.

Parameters

  • arg_name (str) – Name of the input to add options for.

  • description (str) – A human-readable description of the input arg_name

Example

Define an operation and attach a description to its x input argument.

>>> @OperationPlugin
>>> @describe_input('x', 'The value to square.')
>>> def square(x: int = 100) -> int:
>>>     return x**2
@xicam.plugins.operationplugin.describe_output(arg_name: str, description: str)

Decorator to set the description for output arg_name.

This is useful for annotating the parameter with additional information for users.

These annotations are displayed in GUI representations of the operation.

Parameters

  • arg_name (str) – Name of the input to add options for.

  • description (str) – A human-readable description of the output arg_name.

Example

Define an operation and attach a description to its square output.

>>> @OperationPlugin
>>> @output_names('square')
>>> @describe_output('square', 'The squared value of x.')
>>> def square(x: int = 100) -> int:
>>>     return x**2
@xicam.plugins.operationplugin.display_name

Set the display name for the operation.

Display name is how this operation’s name will be displayed in Xi-cam.

Parameters

name (str) – Name for the operation.

Example

Create an operation whose display name is “Cube Operation.”

>>> @OperationPlugin
>>> @display_name('Cube Operation')
>>> def cube(n: int = 2) -> int:
>>>     return n**3
@xicam.plugins.operationplugin.fixed(arg_name, fix=True)

Decorator to set whether or not an input’s value is fixed.

Fixed means that the input’s value is fixed in the context of model fitting.

By default, sets the arg_name input to fixed, meaning its value cannot be changed.

Parameters

  • arg_name (str) – Name of the input to change fix-state for.

  • fix (bool, optional) – Whether or not to fix arg_name (default is True).

  • example (TODO) –

@xicam.plugins.operationplugin.input_names

Decorator to define input names for the operation.

The number of names provided must match the number of arguments for the operation/function.

If not provided, input names will be determined by examining the names of the arguments to the operation function.

Example

Create an addition operation and use the names “first” and “second” for the input names instead of the function arg names (x and y).

>>> @OperationPlugin
>>> @input_names("first", "second")
>>> def my_add(x: int, y: int) -> int:
>>>     return x + y
@xicam.plugins.operationplugin.limits(arg_name, limit)

Decorator to define limits for an input.

Limits restrict the allowable values for the input (inclusive lower-bound, inclusive upper-bound).

Parameters

  • arg_name (str) – Name of the input to define limits for.

  • limit (tuple[float]) – A 2-element sequence representing the lower and upper limit.

Example

Make an operation that has a limit on the x parameter from [0, 100].

>>> @OperationPlugin
>>> @limits('x', [0, 100])
>>> def op(x):
>>>     ...

Make an operation that has a limit on the x parameter from [0.0, 1.0].

>>> @OperationPlugin
>>> @limits('x', [0.0, 1.0])
>>> @opts('x', step=0.1)
>>> def op(x):
>>>     ...
@xicam.plugins.operationplugin.output_shape(arg_name: str, shape: Union[int, Collection[int]])

Decorator to set the shape of an output in an operation.”

Parameters

  • arg_name (str) – Name of the output to define a shape for.

  • shape (int or tuple of ints) – N-element tuple representing the shape (dimensions) of the output.

Example

TODO

@xicam.plugins.operationplugin.opts(arg_name: str, **options)

Decorator to set the opts (pyqtgraph Parameter opts) for arg_name.

This is useful for attaching any extra attributes onto an operation input argument.

These options correspond to the optional opts expected by pyqtgraph.Parameter. The options are typically used to add extra configuration to a Parameter.

Parameters

  • arg_name (str) – Name of the input to add options for.

  • options (keyword args) – Keyword arguments that can be used for the rendering backend (pyqtgraph).

Example

Define an operation where the x input is readonly.

>>> @OperationPlugin
>>> @opts('x', 'readonly'=True)
>>> def op(x: str = 100) -> str:
>>>     return x
@xicam.plugins.operationplugin.units(arg_name, unit)

Decorator to define units for an input.

Associates a unit of measurement with an input.

Parameters

  • arg_name (str) – Name of the input to attach a unit to.

  • unit (str) – Unit of measurement descriptor to use (e.g. “mm”).

Example

Create an operation where its x parameter has its units defined in microns.

>>> @OperationPlugin
>>> @units('x', 'μ'+'m')
>>> def op(x: float = -1) -> float:
>>>     return x *= -1.0
@xicam.plugins.operationplugin.visible(arg_name: str, is_visible=True)

Decorator to set whether an input is visible (shown in GUI) or not.

Parameters

  • arg_name (str) – Name of the input to change visibility for.

  • is_visible (bool, optional) – Whether or not to make the input visible or not (default is True).

Example

Define an operation that makes the data_image invisible to the GUI (when using as_parameter() and pyqtgraph).

>>> @OperationPlugin
>>> @visible('data_image')
>>> def threshold(data_image: np.ndarray, threshold: float = 0.5) -> np.ndarray:
>>>     return ...
class xicam.plugins.operationplugin.OperationPlugin(**filled_values)

A plugin that can be used to define an operation, which can be used in a Workflow.

Note: use the @operation decorator to create an operation type.

At its simplest level, an operation can be though of as a function. Any arguments (parameters) defined in the python function are treated as inputs for the operation. An operation’s outputs are defined by the returned values of the python function.

There are various methods available to help with modifying the operation’s parameters.

For more information on the attributes, see the documentation for their respective method (e.g. for more information on limits, see the limits method documentation).

For an easy way to expose parameters in the GUI, use OperationPlugin.as_parameter in conjunction with pyqtgraph.Parameter.create. Note that only input parameters that have type hinting annotations will be included in the return value of OperationPlugin.as_parameter.

filled_values

Keys are the parameter names, values are the current values for the parameter.

Type

dict

fixable

Keys are the parameter names, values are bools indicating whether or not the parameter is able to be fixed.

Type

dict

fixed

Keys are the parameter names, values are bools indicating whether or not the parameter is fixed.

Type

dict

input_names

Names (in order) of the input argument(s) for the operation. Note that if not provided, input names default to the argument names in the function signature.

Type

Tuple[str, …]

limits

Keys are the parameter names, values are the limits (which are a collection of floats).

Type

dict

opts

Any additional options (kwargs) to be passed to the parameter (useful with pyqtgraph).

Type

dict

output_names

Names (in order) of the output(s) for the operation.

Type

Tuple[str, …]

output_shape

Keys are the output parameter names, values are the expected shape of the output (which are of type list).

Type

dict

units

Keys are the parameter names, values are units (of type str).

Type

dict

visible

Keys are the pareameter names, values are bools indicating whehter or not the parameter is visible (when exposed using pyqtgraph).

Type

dict

disabled

Whether or not the operation is disabled (default is False).

Type

bool

display_name

The name of the operation as it should be displayed to a user.

Type

str

hints
Type

list

input_descriptions

A mapping dict containing descriptions of each named input parameter

Type

dict

output_descriptions

A mapping dict containing descriptions of each named output parameter

Type

dict

Notes

This class formally deprecates usage of the ProcessingPlugin API.

Example

Here, we define a function, then wrap it with the OperationPlugin decorator to make it an operation.

>>> @OperationPlugin
>>> def my_operation(x: int = 1, y: int = 2): -> int
>>>     return x + y
class xicam.plugins.operationplugin.ValidationError(operation, message)

Bases: OperationError

Exception raised for invalid OperationPlugin configurations.

operation

Reference to the operation that failed its validation check.

Type

OperationPlugin

message

Explanation of the error.

Type

str

Workflow Documentation

This documentation provides information on the Worfklow class and its API reference.

If you are new to developing Xi-CAM plugins, it is recommended that you follow the quick-start documentation first.

For more general development resources, see the Resources page.

Note that the examples in this documentation can be run in a python interpreter outside of Xi-CAM (for demonstration purposes). Auxiliary support code to be able to do this is marked with a comment # Only need if not running xicam. When developing within Xi-CAM, you will not need the lines of code marked with that comment.

What Is a Workflow?

In Xi-CAM, a Workflow is represents a sequence of one or more OperationPlugins to execute. Basically, it allows you to process data through some pipeline of operations. Multiple operations can be linked together in a Workflow, provided that the connection between any two operations is compatible (based on inputs and outputs). Execution can be performed asynchronously or synchronously.

Where Is Workflow?
xicam.core.execution.Workflow
What Does a Workflow Look Like?

As mentioned previously, a Workflow can be thought of as a graph-like structure. We can add operations (nodes) and connect them with links (edges).

Example 
from xicam.core import execution  # Only need if not running xicam
from xicam.core.execution import localexecutor  # Only need if not running xicam
from xicam.core.execution import Workflow
from xicam.plugins.operationplugin import operation, output_names

execution.executor = localexecutor.LocalExecutor()  # Only need if not running xicam

# Define our operations
@operation
@output_names("sum")
def my_add(x, y):
    return x + y

@operation
@output_names("square_root")
def my_sqrt(n):
    from math import sqrt
    return sqrt(n)

# Instanciate operations
add_op = my_add()
sqrt_op = my_sqrt()

# Create a Workflow and add our operation instances to it
workflow = Workflow()
workflow.add_operations(add_op, sqrt_op)

# Link the "sum" output of add_op to the "n" input of sqrt_op
workflow.add_link(add_op, sqrt_op, "sum", "n")

# Execute the workflow, sending 1 and 3 as initial inputs to add_op (the first operation)
# This should give us sqrt(1 + 3) -> 2.0.
result = workflow.execute_synchronous(x=1, y=3)
print(result)  # Should be ({"square_root": 2.0},)

In this example, we use an addition operation and a square root operation in our Workflow. We want to add two numbers, then take the square root of the sum.

First, we instanciate our two operation types. This gives us an add_op operation object and a sqrt_op operation object.

Next, we add our operations to the workflow.

We then want to link the operations together so we first add two numbers, then take the square root of the result. We do this by connecting add_op’s “sum” output to sqrt_op’s “n” input.

Now that we have added our operations and connected them as we like, we can run our workflow. In this case, we will use execute_synchronous (there are other methods for execution which will be explained later).

However, if we just were to try workflow.execute_synchronous(), the workflow wouldn’t know what the “x” and “y” inputs are supposed to be for the first operation, add_op.

We can either:

  1. pass in data into the first operation(s)’ inputs when we call an execute method on the workflow

  2. have a GUI widget that exposes the operations through the GUI (such as WorkflowEditor), which can provide values directly to the operations’ inputs

In this example, we used option 1 (for an example of option 2, see the ExamplePlugin’s use of WorkflowEditor in the quick-start documentation). To do this, we passed x=1 and y=3 to our execute_synchronous call, which provided values for the invert operation’s x and y input arguments.

Useful Methods for Modifying the Workflow

Here is a condensed version of the various ways to modify a Workflow’s operation and links. For more information, see the API Reference.

Adding, Inspecting, and Removing Operations

Adding operations:

  • add_operation – add an operation to the Workflow

  • add_operations – add multiple operations to the Workflow

  • insert_operation – insert an operation at a specific index in the Workflow

Inspecting operations:

  • operations – get the operations currently in the Workflow

Removing operations:

  • remove_operation – remove an operation from the Workflow

  • clear_operations – remove all operations from the Workflow

Enabling and Disabling an Operation

It is possible to enable or disable operations. By default, all operations added to a Workflow are enabled. For more information, see the API Reference.

Executing a Workflow

When you execute a Workflow, the operations are executed based on how they are linked together.

There are a few ways to run a Workflow: execute, execute_synchronous, and execute_all.

Synchronous Execution

As we saw in our example earlier, we can use execute_synchronous to run a Workflow as a normal snippet of Python code. When this method is run, the we wait until we get a result back before the interpreter can continue running code.

API Reference
class xicam.core.execution.Workflow(name='', operations=None)

Bases: Graph

Add a link between two operations in the workflow.

Links are defined from an operation’s parameter to another operation’s parameter. This creates a connection between two operations during execution of a workflow.

Parameters

  • source (OperationPlugin) – The operation to link from.

  • dest (OperationPlugin) – The operation to link to.

  • source_param (str) – Name of the parameter in the source operation to link (source of the data; output).

  • dest_param (str) – Name of the parameter in the destination operation to link (where the data goes; input).

add_operation(operation: OperationPlugin)

Add a single operation into the workflow.

add_operations(*operations: OperationPlugin)

Add operations into the workflow.

This will add the list of operations to the end of the workflow.

as_dask_graph()

process from end tasks and into all dependent ones

Returns a tuple that represents the graph as a dask-compatible graph for processing. The second element of the tuple identifies the end node ids (i.e. nodes that do not have connected outputs).

Returns

A tuple with two-elements, the first being the dask graph, the second being the end task ids.

Return type

tuple

attach(observer: Callable)

Add an observer to the Workflow.

An observer is a callable that is called when the Workflow.notify method is called. In other words, the observer will be called whenever the Workflow state changes; for example, links are modified, operations are removed, etc. When notified, the observer is called.

Parameters

observer (Callable) – A callable to add from the Workflow.

auto_connect_all()

Attempts to automatically connect operations together by matching output names and input names.

Makes a best-effort to link operations based on the names of their outputs and inputs. If operation A has an output named “image”, and operation B has an input named “image”, then A “image” will link to B “image”. Outputs and inputs that have matching types in addition to matching names will be favored more for the auto-connection.

If there are no outputs with matching inputs (by name), no links will be added.

Remove all links from the workflow, but preserve the operations.

Remove all links for an operation.

clear_operations()

Remove all operations and links from the workflow.

detach(observer: Callable)

Remove an observer from the Workflow.

An observer is a callable that is called when the Workflow.notify method is called. In other words, the observer will be called whenever the Workflow state changes; for example, links are modified, operations are removed, etc. When notified, the observer is called.

Parameters

observer (Callable) – The callable to remove from the Workflow.

disabled(operation)

Indicate if the operation is disabled in the workflow.

Parameters

operation (OperationPlugin) – Operation to check if it is disabled or not.

Returns

Returns True if the operation is disabled in the Workflow; otherwise False.

Return type

bool

disabled_operations()

Returns the disabled operations (if any) in the workflow.

enabled(operation)

Indicate if the operation is enabled in the workflow.

Parameters

operation (OperationPlugin) – Operation to check if it is enabled or not.

Returns

Returns True if the operation is enabled in the Workflow; otherwise False.

Return type

bool

execute(executor=None, connection=None, callback_slot=None, finished_slot=None, except_slot=None, default_exhandle=True, lock=None, fill_kwargs=True, threadkey=None, **kwargs)

Execute this workflow on the specified host. Connection will be a Connection object (WIP) keeping a connection to a compute resource, include connection.hostname, connection.username…

Returns

A concurrent.futures-like qthread to monitor status. The future’s callback_slot receives the result.

Return type

QThreadFuture

execute_all(connection=None, executor=None, callback_slot=None, finished_slot=None, yield_slot=None, except_slot=None, default_exhandle=True, lock=None, fill_kwargs=True, threadkey=None, **kwargs)

Execute this workflow on the specified host. Connection will be a Connection object (WIP) keeping a connection to a compute resource, include connection.hostname, connection.username…

Each kwargs is expected to be an iterable of the same length; these values will be iterated over, zipped, and executed through the workflow.

Returns

A concurrent.futures-like qthread to monitor status. The future’s callback_slot receives the result.

Return type

QThreadFuture

fill_kwargs(**kwargs)

Fills in all empty inputs with names matching keys in kwargs.

Returns the links connected to the operation given (linked inputs of the operation).

The returned dict represents all operations that are connected to operation. Links are represented as a list of 2-element tuples, where the first element of the tuple is another operation’s output parameter, and the second element of the tuple is operation’s input parameter.

Using keys() will give all of the operations that connect to operation. Using values() will give all of the links from each operation to operation.

Parameters

operation (OperationPlugin) – Operation to get incoming links for (some operation -> operation).

Returns

Returns a dictionary defining all of the links from any connected operations to operation.

Return type

defaultdict

Returns the links connected from the operation given (linked outputs of the operation).

The returned dict represents all the operations that operation connects to. Links are represented as a list of 2-element tuples, where the first element of the tuple is operation’s output parameter, and the second element of the tuple is another operation’s input parameter.

Using .keys() on the returned dict will give all of the operations that operation connects to. Using .values() on the returned dict will give all of the links from operation to each operation.

Parameters

operation (OperationPlugin) – Operation to get outgoing links for (operation -> some operation).

Returns

Returns a dictionary defining all of the links from operation to any connected operations.

Return type

defaultdict

insert_operation(index: int, operation: OperationPlugin)

Insert an operation at a specific index in the workflow.

Parameters

  • index (int) – Index where to insert the operation. 0 will add at the beginning; -1 will add to the end.

  • operation (OperationPlugin) – Operation to insert.

Returns all the links defined in the workflow.

Returns a list of tuples, each tuple representing a link as follows: source operation, destination operation, source parameter, destination parameter.

Note that the links are shown as outbound links.

Returns

Returns a list of the links (defined as outbound links) in the workflow.

Return type

list

notify()

Notify the observers; the observers will be called.

Returns the outbound links for an operation.

Returns a list of tuples, each tuple representing a link as follows: operation, destination operation, operation source parameter, destination parameter.

Returns

Returns a list of the links (defined as outbound links) for operation.

Return type

list

property operations

Returns the operations of this workflow.

Remove a link between two operations.

Parameters

  • source (OperationPlugin) – The source operation to remove a link from.

  • dest (OperationPlugin) – The destination operation to remove a link from.

  • source_param (str) – Name of the source parameter that is defining the link to remove.

  • dest_param (str) – Name of the destination parameter that is defining the link to remove.

remove_operation(operation, remove_orphan_links=True)

Remove an operation from the workflow.

Parameters

  • operation (OperationPlugin) – Operation to remove from the workflow.

  • remove_orphan_links (bool) – If True, removes all links that link to the operation to be removed. If False, does not remove any links for the operation and returns the removed operations links dict (default is True).

Returns

By default (remove_orphan_links is True), returns None. Otherwise, returns the links for the removed operation.

Return type

defaultdict

set_disabled(operation: OperationPlugin, value: bool = True, remove_orphan_links: bool = True, auto_connect_all: bool = True)

Set an operation’s disabled state in the workflow.

By default when disabling an operation, links connected to the operation will be removed (remove_orphan_links would be True). If value is False (re-enabling an operation), then no links are changed.

Parameters

  • operation (OperationPlugin) – The operation whose disabled state is being modified.

  • value (bool) – Indicates the disabled state (default is True, which disables the operation).

  • remove_orphan_links (bool) – If True and value is True, removes the links connected to the operation. Otherwise, no links are changed (default is True).

  • auto_connect_all (bool) – If True, then a best-effort attempt will be made to try to reconnect the operations in the workflow (default is True). See the Graph.auto_connect_all method for more information.

Returns

Returns a list of any orphaned links for an operation that is set to disabled. Default behavior will return an empty list (when remove_orphan_links is True). If enabling an operation (value is False), then an empty list is returned, as no links are changed.

Return type

list

stage(connection)

Stages required data resources to the compute resource. Connection will be a Connection object (WIP) keeping a connection to a compute resource, include connection.hostname, connection.username…

Returns

A concurrent.futures-like qthread to monitor status. Returns True if successful

Return type

QThreadFuture

toggle_disabled(operation: OperationPlugin, remove_orphan_links=True, auto_connect_all=True)

Toggle the disable state of an operation.

By default, when an operation is toggled to a disabled state, any links connected to the operation will be removed.

Parameters

  • operation (OperationPlugin) – The operation to toggle disable state for.

  • remove_orphan_links (bool) – If True, when the operation’s toggle state is toggled to disabled, any links connected to the operation will be removed (default is True).

Returns

Returns a list of any orphaned links for an operation. Default behavior will return an empty list. A non-empty list can be returned when remove_orphan_links is False and the connected operation is toggled to disabled.

Return type

list

validate()

Validate all of: - All required inputs are satisfied. - Connection is active. - ?

Returns

True if workflow is valid.

Return type

bool

Install Xi-CAM

If you haven’t already installed Xi-CAM, follow the installation instructions for your operating system:

Installing Xi-CAM for Linux

Installing Xi-CAM requires a few system components to be installed. After successfully installing these components and Xi-CAM, you will be ready to start developing Xi-CAM plugins!

Install python3

First, ensure that you have python3.8 installed on your system.

Consult your specific distribution’s package manager for installing python3.

Create and Activate a Virtual Environment

Creating a virtual environment allows you to install and uninstall packages without modifying any packages on your system. This is highly recommended.

There are a couple of ways to create a virtual environment:

  1. via the venv module provided with python3

  2. via conda (you will need to install this from anaconda.org or miniconda.org)

In the commands below, we will create a new environment called xicam in your home directory, and then activate the environment.

Once an environment is activated, any packages installed through pip will be installed into this sequestered xicam environment. (If using conda, you can install either with pip or conda.)

virtualenv

If you would like to create a virtual environment, run the following:

cd ~
python3 -m venv xicam
source xicam/bin/actviate
conda

If you would like to create an environment through conda, run the following:

cd ~
conda create -n xicam python=3.8
conda activate xicam
Install Python Qt Bindings

Xi-CAM depends on a GUI application framework called Qt; you will need to install one of the python bindings for Qt (PyQt5 or PySide2) in order to run Xi-CAM.

Make sure that you have activated the xicam environment.

For example, you can install the PyQt5 pip package as follows:

pip install PyQt5
Install the Xi-CAM package

Now that we have activated a new xicam environment and installed PyQt5, we can install Xi-CAM:

pip install xicam

To ensure everything is installed correctly, you can run Xi-CAM as follows:

xicam
Where Do I Go from Here?

You are now ready to start developing plugins for Xi-CAM!

To learn about developing plugins for Xi-CAM, see the Quick Start Guide.

Installing Xi-CAM for MacOS

Installing Xi-CAM requires a few system components to be installed. After successfully installing these components and Xi-CAM, you will be ready to start developing Xi-CAM plugins!

Install python3

First, ensure that you have python3.8 installed on your system.

The quickest way to do this is by downloading and running the python.org installer for python3. The python3.8 macOS 64-bit installer can be found here.

Alternatively, you can install XCode and homebrew to manage multiple versions of python on your system.

Create and Activate a Virtual Environment

Creating a virtual environment allows you to install and uninstall packages without modifying any packages on your system. This is highly recommended.

There are a couple of ways to create a virtual environment:

  1. via the venv module provided with python3

  2. via conda (you will need to install this from anaconda.org or miniconda.org)

Using the Terminal application, we will create a new environment called xicam in your home directory, and then activate the environment.

Once an environment is activated, any packages installed through pip will be installed into this sequestered xicam environment. (If using conda, you can install either with pip or conda.)

virtualenv

If you would like to create a virtual environment, run the following:

cd ~
python3 -m venv xicam
source xicam/bin/actviate
conda

If you would like to create an environment through conda, run the following:

cd ~
conda create -n xicam python=3.8
conda activate xicam
Install Python Qt Bindings

Xi-CAM depends on a GUI application framework called Qt; you will need to install one of the python bindings for Qt (PyQt5 or PySide2) in order to run Xi-CAM.

Make sure that you have activated the xicam environment.

For example, you can install the PyQt5 pip package as follows:

pip install PyQt5
Install the Xi-CAM package

Now that we have activated a new xicam environment and installed PyQt5, we can install Xi-CAM:

pip install xicam

To ensure everything is installed correctly, you can run Xi-CAM as follows:

xicam
Where Do I Go from Here?

You are now ready to start developing plugins for Xi-CAM!

To learn about developing plugins for Xi-CAM, see the Quick Start Guide.

Installing Xi-CAM for Windows

Installing Xi-CAM requires a few system components to be installed. After successfully installing these components and Xi-CAM, you will be ready to start developing Xi-CAM plugins!

Install python3

On Windows, a great way to manage python installations and packages is through Anaconda. Follow their Windows installation instructions, which will install the conda package manager, Anaconda, Anaconda Prompt, and Anaconda Navigator.

  • Anaconda – A package that provides conda and several common python packages

  • Anaconda Prompt – A command line shell for managing conda environments and installing packages

  • Anaconda Navigator – A GUI for managing conda environments and installing packages

Open the Anaconda Prompt program.

Then, create a new environment called xicam. This creates a sequestered space on your system to install xicam and its dependencies without modifying any of your system’s libraries.

Next, activate the environment. This tells the system to use the libraries and applications inside the environment.

conda create -n xicam python=3.8
conda activate xicam
Install Python Qt Bindings

Xi-CAM depends on a GUI application framework called Qt; you will need to install one of the python bindings for Qt (PyQt5 or PySide2) in order to run Xi-CAM.

Make sure that you have activated the xicam environment.

In your open Anaconda Prompt window, install the pyqt conda package as follows:

conda install pyqt
Install the Xi-CAM package

Now that we have activated a new xicam environment and installed pyqt, we can install Xi-CAM using a python package management tool called pip. Run the following in your open Anaconda Prompt.

pip install xicam

To ensure everything is installed correctly, you can run Xi-CAM as follows:

xicam
Where Do I Go from Here?

You are now ready to start developing plugins for Xi-CAM!

To learn about developing plugins for Xi-CAM, see the Quick Start Guide.

Copyable Instructions

Anaconda Prompt:

cd ~
conda create -n xicam
conda activate xicam

conda install -c conda-forge pyqt

pip install xicam

xicam

Overview

In this guide we will:

  • Explore the main window of Xi-CAM

  • Download and install an Example Plugin

  • Configure a sample catalog so we can load data

  • Explore the Example Plugin

Key Concepts

Here is a quick overview of some concepts that will be explored in this guide. Note that more documentation is available for each of these concepts.

We have one GUIPlugin (ExamplePlugin) - this will be a plugin that you will be able to select and see within Xi-CAM. The layout of the GUIPlugin is defined by a GUILayout.

We have a few OperationPlugins (invert and random_noise) - These plugins are basically functions that take in data and output derived data.

We also need a way to actually run data through the operations. To do this, we have a Workflow (ExampleWorkflow) - this contains linked operations to execute (can be thought of like a pipeline).

Looking at Xi-CAM’s Main Window

Let’s look at what the main window in Xi-CAM looks like first:

Xi-CAM main window after loading.

The main window of Xi-CAM after it has finished loading.

When Xi-CAM finishes loading, we see the window as shown above. Any installed GUIPlugins will be visible (and selectable) at the top (note that you will probably not have any installed yet).

We can also see some of the default widgets provided:

  • a welcome widget in the center of the window

  • a preview widget in the top-left (lefttop) of the window, which shows a sample of selected data in the data browser widget

  • a data browser widget on the left of the window, which can show available databroker catalogs

Quick GUILayout Overview

We mentioned the terms center, lefttop, and left above. These correspond to positions in a GUILayout. Here is a quick overview of how the Xi-CAM main window is organized:

Layout of Xi-CAM's main window

The layout of Xi-CAM’s main window.

You can see that the layout of Xi-CAM follows a 3x3 grid, where each section is named according to its orientation in relation to the center of the window.

(Note that any GUIPlugins you create will have one or more of these GUILayouts).

Xi-CAM Menu Bar

At the top of the main window, there is a menu bar that contains some helpful items.

In the File item you can find Settings for Xi-CAM. This includes things like:

  • Logging configuration - where to find the log files, what type of logging record…

  • Theme - change the appearance of Xi-CAM

  • Device settings - allows managing different devices (detectors) (if you have Acquire or SAXS installed)

In the Help item you can find a link to the Xi-CAM documentation, a way to contact the development team, and versioning / licensing information for Xi-CAM.

Download and Install the ExamplePlugin

Now that we have looked at the main window and its layout, let’s download the Example Plugin.

cd ~
git clone https://github.com/Xi-CAM/Xi-CAM.ExamplePlugin
cd Xi-CAM.ExamplePlugin
What’s Inside the ExamplePlugin Repository

The repository will contain the following:

Contents of ExamplePlugin repo

The contents of the ExamplePlugin repo when you clone it.

At the top there are a few files and directories:

  • setup.py - describes how to install this as a python package; also used to register plugins (via entry points).

  • configure - special directory for this example, helps set up a catalog

  • xicam - directory that acts as a python namespace package

In xicam, there is a exampleplugin subpackage that contains:

  • __init__.py - makes exampleplugin a python package; also exposes the ExamplePlugin class

  • exampleplugin.py - module that contains the ExamplePlugin GUI plugin

  • operations.py - module that contains the example OperationPlugins

  • workflows.py - module that contains the example Workflows

How Do I Install the Example Plugin?

So far, we have only downloaded the Example Plugin - we still need to install it so Xi-CAM can find it and load it.

We can install downloaded plugins using a pip editable install:

pip install -e .

This uses Python’s entry points mechanism to register plugins for Xi-CAM to see.

Exploring the Example Plugin Interface

When you run xicam, you should now see the Example Plugin available at the top right of the main window.

Select it and you should the Example Plugin layout:

Annotated layout of Example Plugin

The Example Plugin. Uses a CatalogView, DynImageView, and WorkflowEditor as widgets in its layout.

In the center, we have a CatalogView that will be used to display loaded data. On the right, there is a WorkflowEditor that shows the operations in the workflow and allows for running the workflow. At the bottom, there is a DynImageView, which will be used to display the results data.

How Do I Load Data?

Now that we have the Example Plugin installed, we need to have data to load into it.

For purposes of this guide, we will be configuring a catalog called “example_catalog.”

For more information, see the Bluesky DataBroker documentation.

Configuring a Catalog

There is a configure/ directory in the repository we cloned. This contains a catalog configuration file, a msgpack catalog, and a script.

Feel free to inspect the script before you run it; it will attempt to set up a msgpack catalog source for Xi-CAM to use:

cd configure
python setup_catalog.py
cd ..
Loading a Catalog from the Data Resource Browser

Now that we’ve configured the catalog, let’s make sure that Xi-CAM can see it.

When loading a catalog into Xi-CAM, you must have a GUIPlugin active. Let’s select our “Example Plugin.”

Look at the Data Resource Browser on the left hand side of the window. The Data Resource Browser gives us access to two different types of data browsers by default:

  • a bluesky browser for catalogs (adapted from work done by NSLS-II)

  • a local file browser

After configuring our example catalog, the bluesky catalog browser should have the text “example_catalog” in the Catalog drop-down box.

Notice that it also has two text inputs, Since and Until. Our example catalog was created in the beginning of 2020. In order to see the data (catalogs) our “example_catalog” contains, we need to change the Since text input.

Change it’s value to “2020-01-01”. This will now look for any data that was created since the start of 2020. After making this change, the example_catalog will be re-queried for data created within these new dates.

You should see a catalog show up in the table below with the id 349497da. If you single-click the row in the table to highlight it, more information and a preview of the data should be shown as well. You can then open it with the “Open” button.

Here we see catalog 349497da in the DataResourceBrowser. It has one stream (primary) with 10 events in it. A preview shows the first frame of the data, and the opened data appears in the center.

You should see Clyde the cat loaded into the center CatalogView.

Running a Workflow

Our Example Plugin has one internal workflow, the ExampleWorkflow. The ExampleWorkflow contains two OperationPlugins (operations):

  • invert - inverts its input image

  • random_noise - applies random noise to its input image, has a “strength” parameter to define how much noise to apply to the image

This workflow is exposed in the GUI with a WorkflowEditor on the right side of the layout.

Now that we have loaded some data, let’s run our workflow by clicking the “Run Workflow” button.

Resulting image after running workflow in the Example Plugin.

The result data after running the workflow. Note that the color lookup table can be changed by right-clicking the gradient bar.

You should see an inverted picture with some random noise added to it.

Note that you can adjust the amount of random noise by selecting the “random_noise” text in the WorkkflowEditor, then changing the value of “strength” that shows up in the parameter tree above.

Examining the Code

Let’s take a quick look at how the code is implemented for our Example Plugin.

The code for this particular plugin is organized into three modules:

  • exampleplugin.py - Defines the ExamplePlugin (the GUIPlugin)

  • operations.py - Defines two OperationPlugins: invert and random_noise

  • workflows.py - Defines an ExampleWorkflow with the invert and random_noise operations

operations.py

Here we define OperationPlugins (or operation)

An operation can be thought of as a function; input data is sent into the operation, and the operation generates some output with the given input.

When defining an OperationPlugin, we use Python decorators (the @ seen in the code). At the very least, you must provide the @operation and @output_names decorators for an operation.

workflows.py

Here we define an ExampleWorkflow.

We add our two operations to the ExampleWorkflow, then connect them so that invert’s “output_image” value is sent to random_noise’s input image argument.

exampleplugin.py

Here we define the gui plugin ExamplePlugin.

We provide a name for the plugin, which will display as “Example Plugin” in Xi-CAM.

We define our widgets, our layout, and any internal objects we might need (like the workflow) inside of our __init__ method. We connect the WorkflowEditor’s sigRunWorkflow signal to our run_workflow method. This means whenever “Run Workflow” is clicked in the WorkflowEditor, our ExamplePlugin’s run_workflow method will be called.

We also define a results_ready method that will be called whenever our workflow has finished executing its operations. Providing callback_slot=self.results_ready in our execute call sets up this connection for us.

Entry Points

An entry point is a mechanism that can be used to make objects discoverable by a common interface / name.

Xi-CAM Entry Points

In Xi-CAM, you can define entry points and then run pip install -e . in your plugin package directory to register plugins. This allows Xi-CAM to see your plugins when it loads. Entry points are defined in setup.py files, in the entry_points key.

Let’s look at an example repository and setup.py:

setup.py
xicam/
    myplugin/
        __init__.py           - defines MyGUIPlugin (also marks this directory as a Python module)
        operations/ 
            __init__.py       - marks this directory as a Python module
            edge_detection.py - contains edge detection operations (laplace and sobel)
        workflows/
            __init__.py       - (marks this directory as a Python module)
            myworkflow.py     - defines MyWorkflow.py

Here’s what our entry_points might look like in setup.py:

entry_points = {
    "xicam.plugins.GUIPlugin": ["myguiplugin = xicam.myplugin:MyGUIPlugin"],
    "xicam.plugins.OperationPlugin": [
        "laplace_operation = xicam.myplugin.operations.edge_detection:laplace",
        "sobel_operation = xicam.myplugin.operations.edge_detection:sobel"
    ],
}

As seen above, entry_points is a dictionary, where each key is an entry point and each value is a list of objects / types being registered to that entry point.

The syntax is: "entry point name": ["some_identifier = package.subpackage.module:ClassName"].

In this case, we are registering MyGUIPlugin to the xicam.plugins.GUIPlugin entry point. Similarly, we are registering the laplace and sobel operations to the xicam.plugins.OperationPlugin entry point.

Note that Workflows are not registered in this way; they are not Xi-CAM plugins.

Whenever you modify entry points, you must reinstall your package. You can do this by running pip install -e . in your package directory.

When Xi-CAM loads, it will see the xicam.plugins.GUIPlugin entry point key and load in MyGUIPlugin defined (in the value). Similarly, Xi-CAM will see the xicam.plugins.OperationPlugin entry point key and load in the laplace and sobel operations.

More Information

For more information about entry points, see the following:

  • https://entrypoints.readthedocs.io/en/latest/

  • https://packaging.python.org/specifications/entry-points/

  • https://amir.rachum.com/blog/2017/07/28/python-entry-points/

Data ingestion in Xi-CAM

What is an ingestor?

The ingestor design is specified by the Databroker team to provide an entrypoint for data generated external from the Bluesky environment. An ingestor is a Callable that accepts a URI (often a local file path) and yields (name, doc) pairs. The yielded data follows the Bluesky event-model structure (see event-model documentation). Synthesizing these event-model documents is made easier with the RunBuilder (see Bluesky-Live documentation).

Resources

Example Xi-CAM Plugins

Git

  • Try GitHub - Landing page for some introductions and resources about git and GitHub.

  • Git Handbook - An introduction to git and GitHub.

NSLS-II

Useful resources about NSLS-II software that Xi-CAM uses.

Python

Here are a few resources regarding object-oriented programming with Python3. Feel free to look through these or even through resources you find on your own if you are interested.

Qt

Qt is a framework written in C++ for developing graphical user interfaces. PySide2 and PyQt5 are two different python bindings to the Qt C++ API. QtPy is a wrapper that allows for writing python Qt code with either PyQt5 or PySide2 installed.

Xi-CAM uses QtPy to interact with different Python bindings to Qt. QtPy allows you “to write your code as if you were using PySide2 but import Qt modules from qtpy instead of PySide2 (or PyQt5)”. The references below show PySide2 examples and documentation; when writing a Xi-CAM plugin, make sure to use the qtpy modules when importing.

  • PySide2 Documentation - Documentation for PySide2. Since the QtPy API resembles PySide2, this documentation is helpful for looking up python Qt modules and classes that you may use.

  • PyQt5 GUI Tutorial - Introductory tutorial for learning the basic concepts of Qt. Note: this tutorial is written for PyQt5, remember to import from qtpy instead of PyQt5 or PySide2 when writing code for Xi-CAM.

  • PySide2 Simple Clickable Button - Short tutorial that describes the concept of signals and slots in Qt and shows how to create a button that responds to clicking.

  • PyQtGraph - Documentation for the pyqtgraph package, which relies on Qt and provides basic data visualization (plotting) and various widgets (helpful for writing Xi-CAM GUIPlugins).

Indices and tables