Improved Analyses loading
I finally succeeded in speeding up loading analyses from the database. In reality retrieval from the database
is quite rapid (depending on network connection) but converting to DVCIsotopeRecordViews was time consuming. One
earlier optimization was to limit the number of progress.change_message calls. change_message eventually
calls Qt.QApplication.process_events which takes ~80ms each call. So loading of analyses was already penalized
80ms because of the progress bar. I will continue to investigate the progress window but a more significant speed up
was achieved by eliminating the concept of an IsotopeRecordView. (eventually this model should be extended to the
other record view classes).
So… the major change was to keep the session open when a SessionCTX is exited and to temporarily disable
expire_on_commit when retrieving analyses. Since the analyses do not change in the database it is safe to
disable expire_on_commit
Sample management using Sample and Sample Prep tasks
I recently added two new tasks to the Entry plugin: 1) Sample, for the initial entry of samples and 2) Sample Prep.
Sample is pretty straight forward. It provides ComboBox fields for Principal Investigator, Project, Material and Sample.
An additional field was added to Material, grainsize. So material names such as San 25-32mesh are no longer necessary.
Sample Prep is a glorified lab note book. Its organized by Session and Worker. Session is a simple way of group a set of samples. Worker is the person assigned to work on set of sample.
Day5
Day 5 of daily blogging. Yesterday I worked on the XLSIrradiationLoader and the associated tests. All tests are
passing and the process appears to work correctly. My main concern is the automatic labnumber generation. There are
some boundary cases that may cause unexpected results. These can be avoided by careful configuration but it is not
wise to leave this to users. They will only be confused and incapable of using this feature correctly. First, duplicate
labnumbers are only skipped and not properly handled, i.e. if a duplicate is encountered a sophisticated mechanism for
1) jumping ahead to the first available labnumber or 2) canceling is required. Second, if the
base_irradiation_offset is less than the total number of positions for a given irradiation, the autogenerated
labnumber for the next irradiation/level will be a duplicate. Third, using different starting points for Monitors and
Unknowns is not currently available.
There is the option to explicitly set the labnumbers in the Excel file. Perhaps the best solution to labnumber numbering is to do a four step process. 1) Add all the irradiations, levels and positions in the import file 2) generate labnumbers for all positions but don’t save 3) display the table of generated labnumbers, identifying conflicts and allow the user to make fixes 4) save the labnumbers.
I also started work on the a class to export irradiation information from Pychron to MassSpec. Unittests have been developed and the first few steps of the process are tested and passing, but more work is required for exporting the production ratios and irradiation positions.
Today I will continue to work on the XLSIrradiationLoader, addressing mainly the auto labnumber generation. I will
also start to work on the manual entry process. The current model is to use something similar to the Experiment editing
process, where there is a separate editor pane for interacting with the level table. Using the editor pane allows for
bulk editing and selecting existing projects, samples, and materials from the database.
Optimization
Speeding up the analysis loading process would greatly enhance usablity and the UX.
Many attempts have been made to optimize this process with varying degrees of success. A new branch feature/new_db was created to explore database optimizations that could speed up db queries. Initial testing with a db of ~35k analyses show ~40% increase in speed. These numbers should be verified. The new db migration is tricky so effort should be put into optimization without radical database changes.
From initial testing it appears the major bottle neck occurs when construction the Isotope objects for each analysis; ~50 ms per analysis. It is currently unclear how isotope construction time can be minimized. Traits overhead maybe considerable but it doesn’t seem likely the Traits dependency can be removed.
Low hangin fruit
- temporally cache irradiation data. When loading multiple analyses use a irradiation cache. Maintaining
the cache outside of a session may be necessary as sqlachemy does internal caching. This suggests that
little to no performance gain will be achieved by discarding the irradiation cache at the end of
make_analyses. - optimize the sync process. optimize DBAnalysis.sync. investigate all aspects of the sync process except isotope construction
write a benchmarking module for systematic analysis of the changes and their effect.
Day3
Day 3 of daily blogging.
Yesterday there was much discussion about usablity that I was not to pleased with. It is a constant battle to help people understand something that they neither have the capacity or desire to comprehend.
A logger pane was added to the experiment window in an effort to present more information to the user easily. This is a completely redundant pane as all the information is displayed in the Logger Window. The logger pane was added to bring the info and warning messages to the forefront. However I do not suspect this in itself will assuage any user issues as no one has adequate awareness of what is happening to read or comprehend the logger messages.
Another Tab should be added to the logger pane for Critical messages. I added
a method to logger_setup called set_exception_handler which hooks into
sys.excepthook. set_exception_handler takes one optional argument func.
func should be a callable that takes three arguments exctype, value, traceback,
where exctype is the Exception Class exctype is the Exception object and traceback
is a traceback object (use traceback.format_tb or print_tb to format the traceback object
for printing or logging). If no func argument is supplied a default exception_hook is
used. The default simply logs the exception as a critcal log message and calls the
original sys.excepthook using sys.__excepthook__
Some other mechanism could be added for presenting additional information to the user. Users apparently are having a difficult time interpreting failure states (in my opinion because of a lack of desire, experience and capacity to understand). Typically users do not recognize 1) what the problem is or 2) what solution to use. All of the failure states should be laid out in the documentation but again since no one uses the documentation, this is unrecognized work. An effort, however, to make info, confirmation, and warning messages more descriptive, less esoteric and jargony is desirable.
There are three cases in which users appear to have the most trouble 1) program launch 2) experiment launch 3) intra experiment failures. In all cases a dirth of information and messages are supplied to the user for diagnosis. In all cases improving the readability of the information would go a long way. The location of all the information remains a problem and I do not yet have a good solution. One would think the logger window and popup dialogs would be sufficient but 1) users are incapable of navigating and managing multiple windows on the desktop and 2) not interested in learning the meaning, cause, and solution to problems.
Program Launch
Experiment Launch
Intra Experiment Failures
This may seem like alot of whinging about users but this has been a continuing problem and appears that only myself is make a significant effort to rectify. It would be best if users could meet me 10% of the way, which at this moment appears unlikely. End users have explicitly stated they do not want to understand but want to be competent users. This is totally contradictory and makes the teaching and documentation process difficult, time-consuming and ultimately completely unrewarding.
Day2
This is day 2 of daily blogging. Installed pyview_dev1 onto jan yesterday. There are
now three working repositories on Jan, pychron, pychron_2.0.5 and pychron_dev.
The pychron dir is checked out at branch v2.0.4 and should be used for v2.0.4 hotfixes.
pychron_2.0.5 is the current release and requires testing. A dock version of v2.0.5
will not be made until some initial testing has been performed.
Note
Remember to notify everyone about the dock version upgrade when it
happens. pychron_dev is also available and checked out on the develop branch. Use
pychron_dev for pyview fixes and changes for v2.0.6.
There is also now unique "Pychron" folders for each version. Remember this is not hardcoded
but simple each version's preferences points to a different "Pychron" directory. pyview_dev1
is using ~/PyView_dev, pychron_dev/pyexperiment_dev is set to use~/Pychron_dev and
pyexperiment_2.0.4rc.. is hardcoded to use ~/Pychrondata_dev.
Use Preferences>General>Pychron Root to set which "Pychron" directory to use.
Musings
This is the start of an attempt to blog daily. Not much to report so far. Pychron development has slowed with most of
the requested data acquisition features implemented. A large number of features require testing and some features will
require heavy modification before they are ready for release. The currently release is 2.0.5 which should be installed
on Jan and Obama. The development branch is at 2.0.6. A new feature that could mitigate some user confusion (or enhance
it) is the ability to specify the root pychron directory. Previously the root directory was hardcoded and passed in to
the entry_point function via setup_version_id. This process is no longer used and the Pychron root directory
(stored in the python.paths.paths variable as root_dir) is saved in the users preference file. This process introduced
a few issues in terms of when the paths object is built and the launching process but they appear to be resolved. A
change was the users file, used to store the last logged in user and a list of available users, was moved from the
.hidden dir to .enthought. This makes same users file available for all launches on a given computer
instead of dedicated to a specific Pychron root directory. The naming format for the root directory also changed from
Pychrondata to simply Pychron. This removes the naming conflict with the database naming format of
pychrondata[_<name>]. The global initialization variables user_login and multi_user were moved from the
initialization.xml file to General preferences.
Adding labels to the LabBook notes is also now available. To handle the many-to-many relationship between notes and
labels a light-weight SQLite database called labels.db is automatically added to the LabBook directory (i.e.
data/labbook/labels.db). The user can add labels by using the Add Label toolbar button. To apply a label to a
note, 1) open the note 2) double-click the desired label to apply. Active labels for a note are displayed at the bottom
of the note. Active notes also display their color as the label background whereas inactive labels have a white
background and display the label text in their color.
Editing the initialization.xml file via a GUI was added. Access the GUI using MenuBar/Help/Edit Initialization.
This GUI allows the user to enabled/disabled plugins and toggle the boolean value for initialization globals.
Today I will continue working on exporting Irradiation data from a Pychron database to a Mass Spec database. After and/or concurently will be enhancing the manual irradition entry process via pychron and a file based import via Excel spreadsheets
Released v2.0.4
Today I released pychron version 2.0.4. See CHANGELOG for a list of changes. This version is running on the Jan and Obama computers.
Testing of release 2.0.5 will start ASAP.
The development branch is now at v2.0.6
My first blog entry
this is my first blog entry
here is a python code snippet
{% highlight python %}
def main():
print "hello world"
{% endhighlight %}
Rendered as
def main():
print "hello world"1
2
3
4
5
6
7
def show
@widget = Widget(params[:id])
respond_to do |format|
format.html # show.html.erb
format.json { render json: @widget }
end
end
