深海游弋的鱼 – 第 7 页

Python 2.7使用multiprocessing报错PicklingError: Can't pickle : attribute lookup builtin.instancemethod failed

Python 2.7中使用下面代码的时候

import multiprocessing
 
class examplePool:

    def __init__(self):
        self.pool = multiprocessing.Pool()

    def run(self,args):
        print args

    def runProcessPool(self):
        self.pool.apply(self.run,(60,))
        self.pool.close()
        self.pool.join()

if __name__ == '__main__':
    pool = examplePool()
    pool.runProcessPool()

import multiprocessing

class examplePool:

def __init__(self):

self.pool = multiprocessing.Pool()

def run(self,args):

print args

def runProcessPool(self):

self.pool.apply(self.run,(60,))

self.pool.close()

self.pool.join()

if __name__ == '__main__':

pool = examplePool()

pool.runProcessPool()

会报告如下错误：

Traceback (most recent call last):
  File "text.py", line 18, in <module>
    pool.runProcessPool()	
  File "text.py", line 12, in runProcessPool
    self.pool.apply(self.run,(60,))
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 244, in apply
    return self.apply_async(func, args, kwds).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

Traceback (most recent call last):

File "text.py", line 18, in <module>

pool.runProcessPool()

File "text.py", line 12, in runProcessPool

self.pool.apply(self.run,(60,))

File "/usr/lib/python2.7/multiprocessing/pool.py", line 244, in apply

return self.apply_async(func, args, kwds).get()

File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get

raise self._value

cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

这种错误发生的原因是Python 2.7版本中的multiprocessing模块的进程池部分存在BUG，无法正常处理这种写法的代码，据说在Python 3.4版本之后解决了此问题。

目前测试来看，不用Python 2.7中的进程池，而是自己创建进程，自己管理进程的方式，可以比较简单的解决此类问题。

比如代码修改成如下样子，即可正常运行

import multiprocessing
 
class exampleProcess:

    def __init__(self):
        self.process = multiprocessing.Process(target=self.run,args=(60,))

    def run(self,args):
        print args

    def runProcess(self):
        self.process.start()
        self.process.join()

if __name__ == '__main__':
    process = exampleProcess()
    process.runProcess()

import multiprocessing

class exampleProcess:

def __init__(self):

self.process = multiprocessing.Process(target=self.run,args=(60,))

def run(self,args):

print args

def runProcess(self):

self.process.start()

self.process.join()

if __name__ == '__main__':

process = exampleProcess()

process.runProcess()

至于多进程的管理，则只需要使用

from multiprocessing import Process, Queue

1	from multiprocessing import Process, Queue

引入multiprocessing模块的Queue,实现一个简单的生产者，消费者模型即可。

参考链接

python multiprocessing遇到Can’t pickle instancemethod问题

WARNING:libav.swscaler:Warning: data is not aligned! This can lead to a speedloss

在Python中使用libav视频解码的时候，如果需要更改最后输出的视频的宽高，比如如下代码：

width = 240
container = av.open(url)
video_stream = next(s for s in container.streams if s.type == 'video')
for packet in container.demux(video_stream):
   for frame in packet.decode():
        if frame is None:
            break
        frame = frame.reformat(width,height)
        img = frame.to_image()

width = 240

container = av.open(url)

video_stream = next(s for s in container.streams if s.type == 'video')

for packet in container.demux(video_stream):

for frame in packet.decode():

if frame is None:

break

frame = frame.reformat(width,height)

img = frame.to_image()

可能会收到一条警告信息

WARNING:libav.swscaler:Warning: data is not aligned! This can lead to a speedloss

1	WARNING:libav.swscaler:Warning: data is not aligned! This can lead to a speedloss

导致警告的原因是swscaler的缩放的目标尺寸不合适，它预期的大小是16的倍数，这个倍数可以保证swscaler以最高效的方式进行图片的缩放处理。

解决警告的方式就是保证宽高都是16的倍数即可。

参考链接

[swscaler] Warning: data is not aligned! This can lead to a speedloss 的解决方法【FFmpeg】

Ubuntu 16.04下Python打印正在运行的进程的栈信息

Python程序在运行时候，可能由于某些原因导致进程卡住在某行代码上，此时我们需要输出进程中各个线程的栈信息。
此时我们需要使用Python栈工具pstack的协助,项目的工程地址https://github.com/wooparadog/pstack/

具体的用法如下：

$ sudo pip install pstack
$ sudo pstack pid

1 2	$ sudo pip install pstack $ sudo pstack pid

cProfile——Python性能分析工具

Python自带了几个性能分析的模块：profile,cProfile和hotshot，使用方法基本都差不多，无非模块是纯Python还是用C写的。本文介绍cProfile。
继续阅读cProfile——Python性能分析工具

python直接下载图片到内存

1. 使用requests（推荐）

from PIL import Image
from StringIO import StringIO
import requests
Image.open(StringIO(requests.get(url, stream=True).raw.read()))

from PIL import Image

from StringIO import StringIO

import requests

Image.open(StringIO(requests.get(url, stream=True).raw.read()))

2. 使用StringIO

from PIL import Image
from StringIO import StringIO
import requests

r = requests.get("http://a/b/c")
im = Image.open(StringIO(r.content))
im.size

# =======================

from PIL import Image
import urllib2 as urllib
from StringIO import StringIO

fd = urllib.urlopen("http://a/b/c")
im = Image.open(StringIO(fd.read()))
im.size

from PIL import Image

from StringIO import StringIO

import requests

r = requests.get("http://a/b/c")

im = Image.open(StringIO(r.content))

im.size

# =======================

from PIL import Image

import urllib2 as urllib

from StringIO import StringIO

fd = urllib.urlopen("http://a/b/c")

im = Image.open(StringIO(fd.read()))

im.size

3. 使用io.BytesIO

from PIL import Image
import urllib2 as urllib
import io

fd = urllib.urlopen("http://a/b/c")
image_file = io.BytesIO(fd.read())
im = Image.open(image_file)

from PIL import Image

import urllib2 as urllib

import io

fd = urllib.urlopen("http://a/b/c")

image_file = io.BytesIO(fd.read())

im = Image.open(image_file)

参考链接

python直接下载图片到内存

获取当前Python中site-packages的具体存放路径

很多时候，我们系统上安装了好几个版本的Python, 此时，我们往往没办法确定通过pip安装的包会存放到那个目录下的site-packages中，可以通过如下代码获取：

$ python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"

1	$ python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"

在Ubuntu 16.04 LTS系统上，这个输出是存在问题的，执行命令后输出的目录是：

/usr/lib/python2.7/dist-packages

1	/usr/lib/python2.7/dist-packages

实际上，通过pip命令安装的目录有很大一部分被安装到了

/usr/local/lib/python2.7/dist-packages

1	/usr/local/lib/python2.7/dist-packages

目录下。

Ubuntu 16.04 LTS上使用Python3版本的PIP

Ubuntu 16.04 LTS上使用Python2与Python3是共存的，而且默认使用Python2，如果使用Python3则需要明确指定。

1.安装Python3版本的PIP

$ sudo apt-get install python3-pip

$ sudo pip3 install --upgrade pip

$ sudo apt-get install python3-pip

$ sudo pip3 install --upgrade pip

2.安装Python3版本的NumPy

$ pip3 install numpy

1	$ pip3 install numpy

3.安装Python3版本的OpenCV

$ sudo pip3 install opencv-python

1	$ sudo pip3 install opencv-python

注意，目前的Python3版本的OpenCV是不支持cv2.imshow()的，具体查看https://pypi.python.org/pypi/opencv-python，可以看到如下信息：

**Q: Why I can't open GUI windows (``cv2.imshow()``) on GNU/Linux distribution X or on macOS?**

A: Like above, OpenCV was not compiled against GTK or Carbon. Support for these might be added in the future.

**Q: Why I can't open GUI windows (``cv2.imshow()``) on GNU/Linux distribution X or on macOS?**

A: Like above, OpenCV was not compiled against GTK or Carbon. Support for these might be added in the future.

Python 2.7 基于twisted实现简单的web服务器

Python 2.7自带的SimpleHTTPServer默认是HTTP/1.0，导致在投放简单的视频的时候，一般是无法通过FFMPEG进行播放的,主要是HTTP/1.0不支持Content-Range导致无法快进以及视频的Seek操作。并且比较悲剧的是，如该修改成HTTP/1.1协议，默认只有一个连接在处理，导致只要第一个用户不断开，第二个连接基本上连接不上。

网上搜索了以下，找到了用twisted代码来实现比较简单，并且性能还不错的服务器。

本文实例讲述了Python基于twisted实现简单的web服务器，分享给大家供大家参考。具体方法如下：
1.首先是通过PIP安装twisted

$ sudo pip install twisted

1	$ sudo pip install twisted

2. 新建htm文件夹,在这个文件夹中放入显示的网页文件

3. 在htm文件夹的同级目录下,建立web.py,代码如下:

from twisted.web.resource import Resource 
from twisted.web import server 
from twisted.web import static 
from twisted.internet import reactor 
  
PORT = 1234
  
######################################################################## 
class ReStructed(Resource): 
  """"""
  
  #---------------------------------------------------------------------- 
  def __init__(self, filename, *a): 
    """Constructor"""
    self.rst = open(filename).read() 
  def render(self, request): 
    return self.rst 
    
resource = static.File('htm/') 
resource.processors = {'.html':ReStructed} 
resource.indexNames = ['index.html'] 
  
reactor.listenTCP(PORT, server.Site(resource)) 
reactor.run()

from twisted.web.resource import Resource

from twisted.web import server

from twisted.web import static

from twisted.internet import reactor

PORT = 1234

########################################################################

class ReStructed(Resource):

""""""

#----------------------------------------------------------------------

def __init__(self, filename, *a):

"""Constructor"""

self.rst = open(filename).read()

def render(self, request):

return self.rst

resource = static.File('htm/')

resource.processors = {'.html':ReStructed}

resource.indexNames = ['index.html']

reactor.listenTCP(PORT, server.Site(resource))

reactor.run()

执行脚本

$ python web.py

1	$ python web.py

然后浏览器访问：http://127.0.0.1:1234/就可以看到内容了。

参考链接

Python基于twisted实现简单的web服务器

macOS Sierra (10.12.4)下Caffe执行Python代码报告错误“Mean shape incompatible with input shape”

在执行macOS Sierra (10.12.4)下Caffe通过Python接口加载binaryproto格式的均值文件的时候，最后报告错误：

Traceback (most recent call last):
  File "analysis_memnet.py", line 29, in <module>
    detector = caffe.Detector(model_def, pretrained_model, mean=means)
  File "/Users/Source/caffe/distribute/python/caffe/detector.py", line 46, in __init__
    self.transformer.set_mean(in_, mean)
  File "/Users/Source/caffe/distribute/python/caffe/io.py", line 259, in set_mean
    raise ValueError('Mean shape incompatible with input shape.')
ValueError: Mean shape incompatible with input shape.

Traceback (most recent call last):

File "analysis_memnet.py", line 29, in <module>

detector = caffe.Detector(model_def, pretrained_model, mean=means)

File "/Users/Source/caffe/distribute/python/caffe/detector.py", line 46, in __init__

self.transformer.set_mean(in_, mean)

File "/Users/Source/caffe/distribute/python/caffe/io.py", line 259, in set_mean

raise ValueError('Mean shape incompatible with input shape.')

ValueError: Mean shape incompatible with input shape.

这个错误发生的原因是由于memnet提供的均值文件是256*256的，但是提供的配置文件却是227*227的，导致在io.py里面的代码在进行判断的时候发生异常。调整源代码中的python/caffe/io.py里面的代码：

    def set_mean(self, in_, mean):
        """
        Set the mean to subtract for centering the data.

        Parameters
        ----------
        in_ : which input to assign this mean.
        mean : mean ndarray (input dimensional or broadcastable)
        """
        self.__check_input(in_)
        ms = mean.shape
        if mean.ndim == 1:
            # broadcast channels
            if ms[0] != self.inputs[in_][1]:
                raise ValueError('Mean channels incompatible with input.')
            mean = mean[:, np.newaxis, np.newaxis]
        else:
            # elementwise mean
            if len(ms) == 2:
                ms = (1,) + ms
            if len(ms) != 3:
                raise ValueError('Mean shape invalid')
            if ms != self.inputs[in_][1:]:
                raise ValueError('Mean shape incompatible with input shape.')
        self.mean[in_] = mean

def set_mean(self, in_, mean):

"""

Set the mean to subtract for centering the data.

Parameters

----------

in_ : which input to assign this mean.

mean : mean ndarray (input dimensional or broadcastable)

"""

self.__check_input(in_)

ms = mean.shape

if mean.ndim == 1:

# broadcast channels

if ms[0] != self.inputs[in_][1]:

raise ValueError('Mean channels incompatible with input.')

mean = mean[:, np.newaxis, np.newaxis]

else:

# elementwise mean

if len(ms) == 2:

ms = (1,) + ms

if len(ms) != 3:

raise ValueError('Mean shape invalid')

if ms != self.inputs[in_][1:]:

raise ValueError('Mean shape incompatible with input shape.')

self.mean[in_] = mean

调整为：

    def set_mean(self, in_, mean):
        """
        Set the mean to subtract for centering the data.

        Parameters
        ----------
        in_ : which input to assign this mean.
        mean : mean ndarray (input dimensional or broadcastable)
        """
        self.__check_input(in_)
        ms = mean.shape
        if mean.ndim == 1:
            # broadcast channels
            if ms[0] != self.inputs[in_][1]:
                raise ValueError('Mean channels incompatible with input.')
            mean = mean[:, np.newaxis, np.newaxis]
        else:
            # elementwise mean
            if len(ms) == 2:
                ms = (1,) + ms
            if len(ms) != 3:
                raise ValueError('Mean shape invalid')
            if ms != self.inputs[in_][1:]:
                in_shape = self.inputs[in_][1:]
                m_min, m_max = mean.min(), mean.max()
                normal_mean = (mean - m_min) / (m_max - m_min)
                mean = resize_image(normal_mean.transpose((1,2,0)),in_shape[1:]).transpose((2,0,1)) * (m_max - m_min) + m_min
                #raise ValueError('Mean shape incompatible with input shape.')
        self.mean[in_] = mean

def set_mean(self, in_, mean):

"""

Set the mean to subtract for centering the data.

Parameters

----------

in_ : which input to assign this mean.

mean : mean ndarray (input dimensional or broadcastable)

"""

self.__check_input(in_)

ms = mean.shape

if mean.ndim == 1:

# broadcast channels

if ms[0] != self.inputs[in_][1]:

raise ValueError('Mean channels incompatible with input.')

mean = mean[:, np.newaxis, np.newaxis]

else:

# elementwise mean

if len(ms) == 2:

ms = (1,) + ms

if len(ms) != 3:

raise ValueError('Mean shape invalid')

if ms != self.inputs[in_][1:]:

in_shape = self.inputs[in_][1:]

m_min, m_max = mean.min(), mean.max()

normal_mean = (mean - m_min) / (m_max - m_min)

mean = resize_image(normal_mean.transpose((1,2,0)),in_shape[1:]).transpose((2,0,1)) * (m_max - m_min) + m_min

#raise ValueError('Mean shape incompatible with input shape.')

self.mean[in_] = mean

调整完成后，需要重新编译Caffe:

$ make clean
$ make
$ make pycaffe
$ make distribute

$ make clean

$ make

$ make pycaffe

$ make distribute

参考链接

macOS Sierra (10.12.4)编译pycaffe成功后，执行时候崩溃，错误“Segmentation fault: 11”

参照 macOS Sierra (10.12.3)编译Caffe 编译成功 Caffe 后,开始尝试使用 Caffe 的 Python 接口，执行如下命令：

$ make pycaffe

1	$ make pycaffe

编译一切成功，但是当执行

import caffe

1	import caffe

的时候，程序崩溃，提示如下内容：

Segmentation fault: 11

1	Segmentation fault: 11

继续阅读

一	二	三	四	五	六	日
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30