TDDPirate’s Ideas

ישראל זק תרפ”ז - תשע”א

לימדת אותי לכפול ולחלק כשהייתי בגן. אחר כך שכחתי איך מחלקים עד שלמדתי בבית הספר.

השגת לי צעצוע שמחירו היה מיליוני דולר - CDC6400 שהיה המחשב המרכזי של האוניברסיטה העברית בירושלים.

בזכות השבתון שלך, הייתי בארה”ב וזכיתי לטייל שם בתקופה שבה ישראלים עדיין לא הרבו לטייל בחו”ל.

יום אחד הבאת הביתה מכשיר טלפון שבעזרתו הצד השני יכל לשלוח לי מסר על ידי קוד מורס. הנסיונות לא עלו יפה כי אני ואלה שניסו לשוחח איתי טלפונית לא היינו מיומנים בתקשורת מורס.

כשהייתי בתיכון, פתחת חברת ייעוץ גיאולוגי ועזרתי לך בתכנות. לאחר שנים, הנסיון שלך בצד העסקי של הייעוץ הקל עלי את המעבר מעובד שכיר לפרי-לאנסר עצמאי.

ליווית אותי בתקופה שבה התחלתי לנהוג ברכב, מה שהקל עלי את ההשתלבות בתרבות הכביש הישראלית.

כשכתבתי את עבודת המסטר שלי, קבלתי ממך עצה להוסיף לה רשימת שאלות להמשך המחקר.

אתה ואמא ניהלתם אורח חיים שלא הדגיש את החומרנות וצריכת הראווה. המותרות שלכם היו לטייל כמעט בכל חור נידח בעולם. לכן גדלתי בתחושה שכסף לא יהיה מה שימנע ממני להשיג מה שבאמת חשוב בחיים. היה פתרון תקציבי ללימודים בקולג’ גלאודט כך שיכלתי לבחור לא ללמוד שם לא בגלל שיקולים כספיים. עזרתם לי לרכוש דירה מיד כשידעתי איפה אגור. בחרתי לא לעבור ניתוח שתל שבלול אבל דאגתם למימון, כך שאם הייתי רוצה, הייתי יכול לעבור ניתוח זה - גם בתקופה שלפני שהניתוח נכלל בסל הבריאות.

פרויקט הטלכתבים, להנגשה לחרשים של רשת הטלפונים לפני יותר מ-25 שנה, נזקק גם לכל מיני קשרים עם כל מיני בעלי השפעה. פעלת רבות מאחורי הקלעים. אצלנו מקובל להתגאות כשלא נזקקים לקשרים בשביל להצליח. אבל זה היה פרויקט שבשבילו זו היתה מצווה לנצל כל קשר שאפשר ולהפעיל כל השפעה פוליטית שצריך.

לאחר שנים קראתי את ההיסטוריה של הטלכתבים בארץ שבה הכל התחיל - ארה”ב - והתחושה שלי היא שבסך הכל בישראל הכל הלך לנו יותר בקלות מאשר ליוזמי הטלכתבים בארה”ב.

בניגוד להרבה אנשים אחרים, לקחת על עצמך את האחריות לאפשר ולהקל על התקשורת איתי. אני לא זוכר שאי פעם תיקנת לי את הדיבור, ואני זוכר פעם שאמרת לי שכשאני מרגיש בנוח אני מדבר יותר ברור. השקעת את המאמץ בלימוד ואימונים בדיבור מרומז ותמיד היה לי קל להבינך כשדיברת אליי, פרט כמובן לשבועות האחרונים שבהם כבר היית עייף מדי בשביל זה.

אני זוכר שהיו נסיבות שונות בילדותי שבהן לא הבנתי מה הולך ומה רוצים ממני - עד שבאת והסברת לי בבהירות מה הולך שם.

מאז שהטלכתב נכנס לחיינו, היית שרות הממסר שלי, לתיווך ביני ובין אנשים אחרים שהייתי צריך להשיגם בטלפון. לשמחתי הצורך שלי בשרות הממסר הלך וירד עם השנים, עם שיפור הנגישות התקשורתית של הציבור הכללי, הודות למסרונים ולדואר אלקטרוני.

תודה לך, אבא.

Israel Zak 1927-2011
You teached me to multiply and divide when I was at kindergarten age. Later I forgot how to divide until I relearned it in school.
You obtained for me a toy which costed multimillion dollars - CDC6400, which was the central computer of Hebrew University in Jerusalem.
Thanks to your Sabbatical leave, I was in USA and got to travel there at a time in which Israelis still didn’t travel abroad as much as they do today.
One day you brought home a telephone which had the means to let the other side pass a message to me by Morse code. The experiments were not successful because I and my conversation mates were not proficient in Morse communications.
When I was at high school, you opened a geological consulting company and I helped you in software development. Few years later, your experience with the business side of consulting eased my transition from employee to self-employed freelancer.
You escorted me when I started to drive a car, making it easier for me to integrate into the Israeli road culture.
When I wrote my M.Sc. thesis, you advised me to add a list of questions for further research.

You and mother had lifestyle, which did not emphasize materialism or conspicuous consumption. Your luxuries were to travel to almost every distant hole in the world. Therefore I grew up with the feeling that money will not be what’ll prevent me from achieving whatever is important in one’s life. There was a financing solution for my studies in Gallaudet College so I could choose not to study there not due to financial considrations. You helped me buy an apartment as soon as I knew where I’ll live. I chose not to undergo cochlear implantation but you secured financing for this, so that if I wanted, I’d be able to get this operation - even before the operation was included in the Health Basket.

The TDD Project, for making the telephone network accessible to the deaf more than 25 years ago, needed also all kinds of pull with all kinds of people with influence. You did a lot behind the curtains. It is our custom to be proud when pull is not needed in order to succeed. However it was a project for which it was mitzvah to exploit every exploitable connection and pull every available rope. Several years later I read the history of the TDDs where everything started - USA - and my feeling is that overall it was easier for us than for the TDD innovators in USA.

Unlike several other people, you accepted the responsibility to facilitate the communication with me. I do not remember any occasion in which you corrected my speech, and I remember that once you told me that when I feel relaxed, I speak more clearly. You invested the effort into learning and practicing cued speech and it was always easy for me to understand you when you addressed me, except of course for the last few weeks during which you were too tired for this.
I remember various circumstances during my childhood, in which I did not understand what was going on and what people want from me - until you arrived and explained clearly what is going on there.
Ever since the TDD entered our lives, you were my relay service, to relay messages between me and other people whom I needed to contact by telephone. Happily, my need for the relay service was lessened over the years, with improvement in telecommunications accessibility of the general population, thanks to SMS and E-mail.

Thank you, father.

Time has passed since the Nov. 2007 council elections (with no publicity to the results of the pilot of computerized elections), and since the Dec. 2008 Taldor fiasco.

And it turns out that in their infinite wisdom, the Israeli government decided to continue to cater to the impatient, “want to know the results right NOW!!!” segment of the electorate in future elections.  Having learned from the experience of other countries, a system based upon near-field RFID tags was chosen for development and deployment.

However, according to http://www.eng.tau.ac.il/%7Eyash/evoting-relay-rfid2010.pdf, the chosen system suffers from serious vulnerabilities.

It seems that the real agenda of the politicians here, like everywhere, is still to get the populace to accept ballot systems, which allow the elections to be rigged.

Suppose you have been contacted by someone with a great Web 2.0 idea and he wants you to join his startup.
You need to know whether he knows what he is talking about.
The following checklist may help you tell the clueful apart from clueless.
I hope people will be able to contribute advice concerning each item in the checklist as well as more items I missed.

1. Dealing with bad content:
   • Spam
   • Trolling
   • Off-topic user-contributed content
2. Vandalism (and in general, content backup/restore).
3. Legal:
   • Acceptable use guidelines
   • Copyright violations and other issues
4. Content ownership/lockdown policies - will a contributor be able to export his contributions into file/s in his own PC?
5. How will the network effect be overcome (if another Web 2.0 site already exists serving the same need, how to get people to use your Web site instead of the other site, if they already have stuff).
6. Business model (i.e. how to actually get people to pay for the stuff).
7. Scaling with demand (nowadays, thanks to cloud computing services availability, the required scaling is not that of servers but that of customer service personnel and maybe other critical resources).
8. Are there standards (such as XML schema) relevant to the kind of content to be served by the site?

If what Alan Carter says in his The Programmers’ Stone blog is right, then the way our society treats software developers is like requiring surgeons to operate in non-sterile theaters, with inadequate equipment and without enough help from other doctors and nurses.

Yesterday, the primaries election of the Labor party in Israel was aborted and postponed to a later date due to serious malfunctioning of the voting machines deployed for this purpose.

The provider of those machines was Taldor LTD.

Sources (in Hebrew):

• Primaries in Labor Party: why did the voting machines fail?
• Farce in Labor Party: the primaries were postponed due to the failures

I wrote previously about computerized elections.

The other day I stumbled upon the Website http://copyfree.org/, which advocates a software licensing model somewhat similar to LGPL.
See the Website for arguments in favor of this licensing model.

I would, however, stick with GPL/LGPL due to the following reasons:

1. The world has some actors (such as the one whose name starts with M and ends with T) with monopolistic intentions. Copyfree is not strong enough to stop them. GPL (especially its v3) is essential to limit the effects of such actors.
2. Some software developers are not altruistic philanthropists. They expect to be compensated for their software development work. In the case of software which scratches their own itches, an acceptable form of compensation would be enhancements to the software, which fix bugs and - more interestingly - add new features. When wielded by such developers, GPL/LGPL are used much as traditional copyright law is used by creators to get compensated for their creations.
3. In the special case of security software, which should be used by everyone, exemptions can be made on case by case basis. The reasoning is much the same as the one which led USA to release to USSR, in midst of the Cold War, certain technologies for securing atom bombs against accidental detonation. And those were days, in which people were executed for releasing nuclear secrets to the wrong parties (witness the Rosenbergs affair).

An essential part of integration of Scheme (as implemented in Guile) and Python is allowing Python code to call back code implemented in Scheme. It is also desirable to be able to access data and invoke methods on otherwise-opaque objects created and managed in Scheme. The specifics of opaque object access should also be independent of the specific object system being used in the Scheme-written part of the application.

When implementing in PyGuile the proxying scheme, in which PyObjects serve as proxies for SCMs, the following need to be taken into account:

• Data type conversion
• Retention of references as a protection against garbage collection

Data type conversions

A Python callback can be expected to receive positional and keyword arguments, and return a result of any type. Therefore, templates (possibly trivial) for converting between PyObjects (Python data types) and SCMs (Guile data types) need to be associated with each callback.

In the case of objects, we need to associate, with each attribute getter, a template for converting the value from SCM into PyObject. With each attribute setter, a template for converting the value from PyObject into SCM needs to be associated. With each method, which can be invoked on the object, minimum two templates are needed. Three templates should be provided, in case the object needs to be manipulated by an interface, which expects both positional and keyword arguments in the object’s methods.

All the templates needed to work with a SCM (as a callback or as an object) are associated with it when it is wrapped as it is being passed from Guile to Python.

Retention of references

PyObjects, which wrap SCMs, are not expected to be seen by Guile’s garbage collector. Therefore, we need a mechanism for protecting SCMs referenced by PyObjects.

Due to efficiency considrations, Guile’s scm_permanent_object, scm_gc_protect_object or scm_gc_unprotect_object should not be used on every SCM passed to Python. The solution is to create a set object in Guile, protect it using scm_permanent_object (a single call) and then register in it all wrapped SCMs. When a wrapping PyObject’s __del__ function is invoked, one of its actions is to remove the corresponding SCM object from the set. The set will be implemented using a standard hash table, whose keys will be indexes and the data - the SCMs themselves.

You’ll find it in http://www.zak.co.il/a/itches/pyguile.
DISCLAIMER: the software is still at beta level, and has never been tested outside of my own PC.

There is no 1:1 mapping between Scheme and Python data types. As a consequence, there are several cases, in which PyGuile has to guess how would the user like to have the arguments and result of a Python function converted. Instead of guessing, we would like to empower the user to be explicit about the kinds of conversion which he wants.

The following is a census of ambiguous data conversion cases, which I identified.

1. Scheme pair ->
   • Python 2-Tuple
   • Python 2-List
2. Scheme list ->
   • Python Tuple
   • Python List
   • Nested tree of pairs (2-Tuples or 2-Lists)
3. Python 2-Tuple or 2-List ->
   • Scheme pair
   • Scheme list
   • Scheme rational (if the Pythonic data structure consists of two integer values)
4. Scheme alist (association list) ->
   • Python Dict
   • Python Tuple/List of 2-Tuples
5. Python string ->
   • Scheme string
   • Scheme symbol
   • Scheme keyword

   Additional considrations:

   • Case sensitivity of symbols and keywords
   • String representation of keywords in Guile has leading dash - to retain or remove it in the Python side of affairs?
6. Python 1-character string ->
   • Scheme char
   • Scheme string

   Additional considration: utf-8 encoded glyph is a sequence of few characters.

7. Python int ->
   • Scheme int
   • Scheme bignum
   • Scheme char
8. Python None -> One of several possible values: ‘(), #f, SCM_EOL, ‘*None* or another custom Scheme value.
9. Python (),[],{} ->
   • Scheme ‘()
   • SCM_EOL
   • Custom Scheme value
10. Scheme ‘() ->
    • Python ()
    • Python []
    • Python {}
11. SCM_EOL ->
    • Python (),[],{}
    • Python None
    • Custom Python value
12. Scheme rational ->
    • Python Float
    • 2-Tuple of Python Ints
13. Scheme exact/inexact flag in numerical values - if and how to represent it in the Python side of the application?
14. Giant data structures with sparse access needs - lazy vs. eager conversion
15. Exception objects
16. Objects of certain classes (vectors, ports, functions, images, etc.)

There is also the separate issue of string encoding/decoding, with which we deal by mandating that anything passing between Scheme and Python has to be utf-8 encoded.

One of the goals of PyGuile is to make it efficient to invoke Python library functions from Guile. Therefore, efficiency of conversion of function arguments and results is critical.

When there are no user hints, the following inefficiencies occur:

1. PyGuile has to make a default (and possibly sub-optimal) choice when encountering one of the above ambiguous cases. Then the script using the data has to reformat it to match the data format to its actual needs.
2. PyGuile has to identify the data type of each datum. The present implementation does not go into the internal representation of Guile (SCM) and Python (PyObject) objects, therefore PyGuile has to test for various data types one by one, until one of them matches the argument.
3. Sometimes a Python procedure needs to do no processing on one of its arguments. The argument’s value needs only to be passed around as a pointer, or to be inserted into the right place in a result data structure. In such a case, it is desirable to use the most efficient conversion possible i.e. wrap/unwrap opaque objects. This is a generalization of the case of giant data structures with sparse access needs.

Therefore, when performance is critical, hints from the user would help not only to disembiguate the conversion process but also to speed it up.

The user hints will be implemented as follows.
With each function (Python function invoked from Guile, or Guile function invoked from Python) we associate two (possibly degenerate) signatures. One signature will contain the hints for converting the function’s arguments. The second signature will hint how to convert the function’s result. The signatures are Scheme lists, whose leaf nodes are symbols denoting conversion functions.

Chris Jester-Young, in his answer to my question in Stackoverflow, proposed the following function for traversing two corresponding tree structures, and applying the functions in one of them to data in the other one.

  (define (map-traversing func data)

    (if (list? func)

        (map map-traversing func data)

        (func data)))

Using it requires unquoting. Example:

  (map-traversing `((,car ,cdr) ,cadr) '(((aa . ab) (bb . bc)) (cc cd . ce)))

Our implementation will differ from the above in details, as the signatures’ leaf nodes do not denote proper Scheme functions.

Every project, including software development projects, needs an identity. It needs a definition of its boundaries. It has to be clear about what is inside it and what is outside of it.

Without such a definition, the project would try to be too many things for many people, and as a result, its products would not be really useful for anyone.

A project’s identity makes it easier and faster to make design and trade-off decisions.

Given the above trite introduction, and given that there is a list of goals for the PyGuile project, a list of non-goals is needed as well and here it is.

1. Theoretical academic purity - attempt to convert every data type from Guile to Python and vice versa, and to support the whole range of values assumed by each data type.
2. Ability to mix code snippets from both Scheme and Python in the same source code file.
3. Invocation of machine language libraries (static or DLLs) - for this purpose, there are already existing tools (SWIG and PerlXS).
4. Framework for making it easy to add support for interoperation with yet another scripting language.

There are also some goals, which are low priority and I do not plan to shed tears if they prove to be impossible to achieve without significant effort:

1. Thread-safety
2. Tail recursion support