Analysing-lib-python-errors.md

+The lib-python test runs are a little daunting when trying out a new version of PyPy. The test runner aggregates tests across files, which makes it hard to detect systematic errors like new attributes or features that have not been implemented. Here are a few helper scripts that assume you have downloaded the stdout output from the `python testrunner/lib_python_tests.py` step into `/tmp/stdout.txt`
+
+- Get the total tests run/errored/failed
+  ```
+  import re
+  fid = open('/tmp/stdout.txt')
+  res={'Ran': 0, 'failures':0, 'errors':0, 'skipped':0}
+  for line in fid:
+      pats = re.findall('(^Ran \d+|failures=\d+|skipped=\d+|errors=\d+)', line)
+      for p in pats:
+          if p.startswith('Ran'):
+              k,v = p.split(' ')
+              res[k] += int(v)
+          else:    
+              k,v = p.split('=')
+              res[k] += int(v)
+  print(res)
+  ```
+
+- Rank the top error messages.
+  ```
+  grep '^[A-Za-z]*Error' /tmp/stdout.txt |sort |uniq -c |sort -n
+  ```
+
+- Find missing attributes that need to be implemented
+  ```
+  grep '^AttributeError' /tmp/stdout.txt |sort |uniq -c |sort -n
+  ```
\ No newline at end of file