Monday, June 11, 2018

Read the proofs!

Now and then one of my friends asks me for recommendations for books to use to re-engage with mathematics.  I am always happy to help and I try to come up with recommendations that will be "goldilocks hard" but also interesting and fun for the person who is asking.  I also selfishly think a little about what it will be like for me to talk about the book's content with my friend.

I look for books that motivate the material well, but also actually prove things and I always make a point of telling my friends that they have to read and understand the proofs, ideally well enough that they can reproduce them.  Today I asked myself why I always make such a big deal of this.  Here's what I came up with:

  1. Mathematics is a progression of ideas, expressed in arguments that prove results.  The results are often useful in solving problems, but the arguments are really where the mathematics is.  If you understand the arguments, you will be able to do problems - often in many different ways.
  2. Proofs show exactly what is needed to get results.  They show you, therefore, exactly where the limits are in applying the results.  When you understand a proof, you know exactly what you have to work with.
  3. Proofs are rigorous and often hard.  If your goal is to be able to apply mathematical concepts and hone your mathematical intuition, working through hard proofs is much better training than just practicing algorithms.  
  4. What you can prove in mathematics is what you know.  Results that you can apply, but don't understand the proofs for, are what Plato would call "true opinions," not knowledge.  Not grounded in proof, your understanding of algorithms and their valid application might "fly away" at any time (See Plato's Meno 97-98, for a delightful account of this).
  5. Go back to 1.  The beautiful thing about studying mathematics "for fun" is that you get to actually do it.  Under time / grade / getting applications done pressure, the mathematics ends up taking a back seat to the applications.  It's like fast-forwarding the show and watching the commercials.  Coming back now, you can actually watch the show.
So read the proofs!

Tuesday, May 8, 2018

Hipparchus 1.3 Released

Today we cut another release of our little Java mathematics library.  Hipparchus is based on a fork of Apache Commons Math.

The 1.3 release includes some nice new features - most notably a redesigned Kalman filter implementation capable of handling nonlinear processes, along with quite a few additional enhancements and bug fixes. 

  • Support was added for complex eigenvalue decomposition.
  • A solver for continuous time algebraic Riccati equations was added.
  • Secondary equations can now update the derivatives of the primary equation in ODE.
  • A matrix decomposer was added to allow configuration of decomposition thresholds independently of the matrix.
  • FastMath was extended to cover the new Java 9 methods so it can still serve as a drop-in replacement for java.lang.Math.

    I had fun fixing some bugs in the stat package and reviewing the Kalman filter design and implementation.  One of the bug fixes, for the Mann-Whitney test implementation, included implementation of exact computation of the sampling distribution of the Mann-Whitney U statistic.  The algorithm for this is nice and simple, but it does not handle the case where there are ties in the data.  I have started working on generalizing the algorithm (to some extent, the definition, actually) to support this.  Comments, patches welcome!

    The release is available on the Hipparchus download page.  The release jars are available from maven central with these coordinates:


    Sunday, October 22, 2017

    Selling the truth

    This month's Significance magazine includes a jarring article about fake news.  As one would expect in Significance, there is interesting empirical data in the article.  What I found most interesting was the following quote attributed to Dorothy Byrne, a British broadcast journalism leader:
    "You can't just feed people a load of facts...we are social animals, we relate to other people, so we have to always have a mixture of telling people's human stories while at the same time giving context to those stories and giving the real facts." 
    Just presenting and objectively supporting debunking factual evidence is not sufficient.  We need to acknowledge that just as emotional triggers are key to spreading fake news, so they need to be considered in repairing the damage.  I saw a great example of that in yesterday's Wall Street Journal.  An article, titled "Video Contradicts Kelly's Criticism of Congresswoman," sets out to debunk the fake news story promulgated by the Trump administration claiming that Florida Rep. Frederica Wilson had touted her personal efforts in getting funding for an FBI building in her district while not acknowledging the slain FBI agents for whom the building was named.  The Journal article could have stopped at the factual assertions that she had not been elected when the funding was approved and that a video of the speech she gave includes her acknowledging the agents.  But it goes on to provide emotive context, describing the Congresswoman's lifelong focus on issues affecting low-income families and her personal connection with Army Sgt. La David Johnson, the Green Beret whose passing ultimately led to her confrontation with the Trump administration.  The details on how she had known Sgt. Johnson's family for generations and that he himself had participated in a mentoring program that she founded provided context for the facts.  The emotive picture painted by the original fake news claim and the administration's name-calling "all hat, no cattle" was replaced with the image of a caring human being.  In that light, it's easier to believe the truth - that Rep. Wilson was gracious and respectful of the fallen agents and their families just as she was of Sgt. Johnson and his family.

    The lesson learned here is that in debunking fake news, "factual outrage" is not enough - we need to focus on selling the truth as the more emotionally satisfying position.  As the Significance article points out, people are drawn to simple explanations and beliefs that fit with what they want to be true.  So to repair the damage of fake news, we have to not just show people that their beliefs are inconsistent with reality - we need to provide them with another, emotionally acceptable reality that is closer to the truth.

    Sunday, July 30, 2017

    Five things I look for in Software Engineers

    I have been interviewing a lot of software engineers recently, as I am leading a new team and looking to expand it.  That has led me to reflect a little on what I am actually looking for.  The following five qualities have been shared by all of the really good, fun-to-work-with developers who I have had the pleasure to work with.

    1. Technical mastery
    Really good developers fully understand what they are doing.  This might sound funny, but unfortunately, it is all too common for people to get things to work by cutting and pasting examples or fumbling through a quasi-random hacking process to arrive at code that "works" without actually understanding how or why (or in fact even if) it works.  There is nothing wrong with experimentation and leveraging experience - and working code - of others.  When really good developers do that, though, they always find their way to full understanding of the technologies and techniques that they are using.  When I interview developers, I always ask them to explain exactly how the solutions that they developed work.  I can usually tell very quickly if I am talking to an individual who masters the technology that they use.  I would much rather have a developer with strong mastery of a small set of technologies than someone whose resume is full of advanced technologies that they don't understand.

    2. Simple mindedness
    In The Humble ProgrammerEdsger W. Dijkstra said "The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague."  Really good developers have a wonderful tendency to keep things as simple as possible, as long as possible.  Fancy constructs, excessive OO complexity, needless external dependencies and exotic algorithms never find their way into their code.  If there is a simple way to do something, that is what they do.  Reading the code of a simple-minded developer is like reading a mathematical paper written by a great mathematician.  If the content is straightforward, the progression is 100% predictable.  You can stop in the middle and scribble out what should come next and then see that is what comes next.  When you get to a difficult part where you have to think, you are happy to see that the author found something so simple that you should have thought of it.

    3. Organizing
    Another one of my favorite Dijkstra quotes is that the art of programming is the art of organizing complexity.  Great developers are organizing forces.  Note that this is not the same as "being organized." It means that they help define problems in a way that they can be solved with simple solutions and they help get contracts, interface boundaries and tests in place so that the teams they are part of can be organized.  The scope of what developers can organize naturally grows as they progress in their careers; but they need to have the drive and ability to be "organizers" from the beginning.  Developers that have to be told how to think about problems are net drags on the teams they are part of.  Good ones are key contributors to their teams arrival at nicely organized approaches to problems.

    4. Fast-learning
    Technology changes so fast and business problems are spread across such a large surface that developers constantly need to learn new things.  And these things are not just programming languages, frameworks or constructs.  Developers need to learn business domain concepts, data science and AI concepts as needed, often in ridiculously short timeframes.  This means that they have to be able to learn very fast.  And they have to be able to do this and immediately exercise their knowledge with a high level of independence.  Its great to be able to learn together and share knowledge with others, but sometimes developers need to figure things out for themselves and good ones have the ability and determination to learn what they need to learn - however hairy it gets - to solve the problems in front of them.

    5. Situational awareness
    Good developers ask about - and clearly understand - the execution context of the code they work on.  If something needs to be thread-safe, they write it that way.  They know what the performance and scalability bottlenecks in and around their code are.  They know about its security context.  They see enough of the larger system that their code is running in / interacting with to ensure that it will be operable, failing fast and loudly when it needs to fail, maintaining invariants that it needs to maintain, and providing sufficient logging / events / monitoring interfaces.  And of course, all of this is validated in unit tests.

    I know some people will say that some of what I have above - especially in 3. and 5. - can't really be expected of "SE's."  These qualities, one might argue, are qualities of architects.  Developers just need to be "feature machines" and architects can worry about how to organize the code and make sure the whole system is operable.  My biggest learning in 30 years of software development is that that is the wrong way to think about it.  Architecture influence and scope of vision naturally increases as developers progress in their careers; but it is part of what they do, every day, from day 1 and the better they do it, the more successful the teams they are part of can be.  And the senior ones - those who might have "Architect" or "Principal" or "Staff" in their titles - need to encourage, cultivate, challenge and be influenced by the design thinking of SEs at all levels.

    Saturday, September 17, 2016

    I Pledge Allegiance...

    Last week,  I heard a segment on NPR about patriotic rituals such as saying the Pledge of Allegiance or standing for the National Anthem.  One statement by a young person was hard for me.  She said she could not recite the Pledge of Allegiance because saying that our republic has "liberty and justice for all" is false.  I get that.  And I certainly respect anyone's right to participate or not participate in saying the Pledge.  What bothers me is that "the republic, for which it stands" is an idea and that idea really does mean liberty and justice for all.  We have never had - and no real nation ever will have - perfect liberty and justice. What we pledge allegiance to is the idea that such a nation can exist.  I bet that 150+ years ago when Abraham Lincoln made his long-remembered remarks at Gettysburg, he knew well that this idea would never be perfectly realized.  He knew that what he himself had done to preserve it was not perfect.  But he really did believe in the idea.  This idea is the source of everything that has ever been good about the United States and everything that will ever be good about us, our children or our children's children.  We cannot abandon this idea because we have not lived up to it - even collectively.  Even if we see endemic and systemic injustice and prejudice, we have to see that as not who we are.  And we need our children to see that.  We don't need to make them say the Pledge or even stand when others do, but we do need them to have faith that this idea really can "long endure" and that there really can be "a new birth of freedom" in the United States.  

    Wednesday, March 16, 2016

    When someone who works for you recommends a book, read it!

    People who work for you have a great perspective on what you need to learn.  Here are some great examples:

    Death by Meeting - in which I learned that my team meetings were,, "suboptimal."   Thanks, Bob!

    The Goal - in which I learned that there was a better way to think about process optimization.  Thanks, Kevin!

    The Phoenix Project - in which I am learning that I did not fully understand the consequences of the previous book.  Thanks, Scott!

    Wednesday, November 18, 2015

    I can see it from where I live

    After seeing it recommended by Dan Pink, I started reading Studs Terkel's classic, Working.  The following quote brought back old memories for me
    There’s not a house in this country that I haven’t built that I don’t look at every time I go by. (Laughs.) I can set here now and actually in my mind see so many that you wouldn’t believe. If there’s one stone in there crooked, I know where it’s at and I’ll never forget it. Maybe thirty years, I’ll know a place where I should have took that stone out and redone it but I didn’t. I still notice it. The people who live there might not notice it, but I notice it. I never pass that house that I don’t think of it. 
    I was lucky to have as my first boss a man who really valued workmanship.  His landscape construction and maintenance business was hard and I could see every day the pressure to cut corners.  But he never did and he got very, very mad when he saw any of us doing shoddy work.  

    I remember a few years later, I was working on an interstate highway construction project.  My job was to break off concrete pipes and parge cement around the gaps between them and the junction boxes where they came together.  I always tried to do a nice job, leaving the box looking like it had been cast as a single piece of concrete.  I remember once having a hard time with one of the pipes and struggling with my coworkers to get the finish smooth.  One of them said, "I can't see it from where I live."  I immediately thought of my first boss, yelling at me once for justifying a sloppy joint by saying that no one would notice it because it was going to be backfilled.  He said, "but I just noticed it and you saw it yourself.  When you go home, you will see it again.  And if you don't see it again, you haven't learned anything from me."