样式转换的工作原理详细实例图解

本文概述

• 导入和配置模块
• 创建显示图像的功能
• 定义样式和内容表示
• 样式和内容的中间层
• 克矩阵
• 运行梯度下降
• 变换图像
• 总变化损失
• 重新运行优化功能
• 最终保存结果

神经样式转移是用于获取两个图像(内容图像和样式参考图像)并将它们融合在一起的优化技术, 因此输出图像看起来像内容图像, 但以样式参考图像的样式”绘制”。

导入和配置模块

开启Google colab

from __future__ import absolute_import, division, print_function, unicode_literals

try:
# %tensorflow_version only exists in Colab.
%tensorflow_version 2.x
except Exception:
pass
import tensorflow as tf

输出

TensorFlow 2.x selected.

import IPython.display as display
import matplotlib.pyplot as plt
import matplotlib as mpl
mpl.rcParams['figure.figsize'] = (12, 12)
mpl.rcParams['axes.grid'] = False
import numpy as np
import time
import functools

content_path = tf.keras.utils.get_file('nature.jpg', 'https://www.eadegallery.co.nz/wp-content/uploads/2019/03/626a6823-af82-432a-8d3d-d8295b1a9aed-l.jpg')
style_path = tf.keras.utils.get_file('cloud.jpg', 'https://i.pinimg.com/originals/11/91/4f/11914f29c6d3e9828cc5f5c2fd64cfdc.jpg')

输出

Downloading data from https://www.eadegallery.co.nz/wp-content/uploads/2019/03/626a6823-af82-432a-8d3d-d8295b1a9aed-l.jpg
1122304/1117520 [==============================] - 1s 1us/step
Downloading data from https://i.pinimg.com/originals/11/91/4f/11914f29c6d3e9828cc5f5c2fd64cfdc.jpg
      49152/43511 [=================================] - 0s 0us/step5. def

检查最大尺寸为512像素。

load_img(path_to_img):
max_dim = 512
img = tf.io.read_file(path_to_img)
img = tf.image.decode_image(img, channels=3)
img = tf.image.convert_image_dtype(img, tf.float32)
shape = tf.cast(tf.shape(img)[:-1], tf.float32)
long_dim = max(shape)
scale = max_dim / long_dim
new_shape = tf.cast(shape * scale, tf.int32)
img = tf.image.resize(img, new_shape)
img = img[tf.newaxis, :]
return img

创建显示图像的功能

def imshow(image, title=None):
 if len(image.shape) > 3:
 image = tf.squeeze(image, axis=0)

plt.imshow(image)
if title:
plt.title(title)

content_image = load_img(content_path)
style_image = load_img(style_path)
plt.subplot(1, 2, 1)
imshow(content_image, 'Content Image')
plt.subplot(1, 2, 2)
imshow(style_image, 'Style Image')

输出

x = tf.keras.applications.vgg19.preprocess_input(content_image*255)
x = tf.image.resize(x, (224, 224))
vgg = tf.keras.applications.VGG19(include_top=True, weights='imagenet')
prediction_probabilities = vgg(x)
prediction_probabilities.shape

输出

Downloading data from https://github.com/fchollet/deep-learning-    models/releases/download/v0.1/vgg19_weights_tf_dim_ordering_tf_kernels.h5

574717952/574710816 [==============================] - 8s 0us/step
TensorShape([1, 1000])

predicted_top_5 = tf.keras.applications.vgg19.decode_predictions(prediction_probabilities.numpy())[0]
[(class_name, prob) for (number, class_name, prob) in predicted_top_5]

输出

Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/imagenet_class_index.json
40960/35363 [==================================] - 0s 0us/step
[('mobile_home', 0.7314594), ('picket_fence', 0.119986326), ('greenhouse', 0.026051044), ('thatch', 0.023595566), ('boathouse', 0.014751049)]

定义样式和内容表示

使用模型的中间层来表示图像的内容和样式。从输入层开始, 前几层激活表示低级表示类似的边缘和纹理。

对于输入图像, 请尝试在中间层匹配相似的样式和内容目标表示。

加载VGG19并在我们的映像上运行, 以确保在此处正确使用它。

vgg = tf.keras.applications.VGG19(include_top=False, weights='imagenet')
print()
for layer in vgg.layers:
print(layer.name)

输出

Download data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg19_weights_tf_dim_ordering_tf_kernels_notop.h5
80142336/80134624 [==============================] - 2s 0us/step

input_2
block1_conv1
block1_conv2
block1_pool
block2_conv1
block2_conv2
block2_pool
block3_conv1
block3_conv2
block3_conv3
block3_conv4
block3_pool
block4_conv1
block4_conv2
block4_conv3
block4_conv4
block4_pool
block5_conv1
block5_conv2
block5_conv3
block5_conv4
block5_pool

# Content layer
content_layers = ['block5_conv2'] 

# Style layer of interest
style_layers = ['block1_conv1', 'block2_conv1', 'block3_conv1', 'block4_conv1', 'block5_conv1']

num_content_layers = len(content_layers)
num_style_layers = len(style_layers)

样式和内容的中间层

在较高的层次上, 对于执行图像分类的网络, 它了解图像并要求将图像作为像素并构建内部插图, 该插图将原始图像像素转换为图像中存在的复杂特征。

这也是卷积神经网络可以很好地泛化的原因：它们可以捕获类内的偏差和定义特征(例如猫与狗), 这些特征与图像输入模型和输出排列标签, 模型无关作为复杂的特征提取器提供。通过访问模型的中间层, 我们可以描述输入图像的样式和内容。

建立模型

定义了tf.keras.applications中的网络, 因此我们可以使用Keras功能API轻松提取中间层值。

要使用功能性API定义任何模型, 请指定输入和输出：

模型=模型(输入, 输出)

给定的函数将构建一个VGG19模型, 该模型返回中间层的列表。

def vgg_layers(layer_names):
""" Creating a vgg model that returns a list of intermediate output values."""
# Load our model. Load pretrained VGG, trained on imagenet data
vgg = tf.keras.applications.VGG19(include_top=False, weights='imagenet')
vgg.trainable = False
outputs = [vgg.get_layer(name).output for name in layer_names]
model = tf.keras.Model([vgg.input], outputs)
return model

style_extractor = vgg_layers(style_layers)
style_outputs = style_extractor(style_image*255)
#Look at the statistics of each layer's output
for name, output in zip(style_layers, style_outputs):
  print(name)
  print("  shape: ", output.numpy().shape)
  print("  min: ", output.numpy().min())
  print("  max: ", output.numpy().max())
  print("  mean: ", output.numpy().mean())
  print()

输出

block1_conv1
  shape:  (1, 427, 512, 64)
  min:  0.0
  max:  763.51953
  mean:  25.987665

block2_conv1
  shape:  (1, 213, 256, 128)
  min:  0.0
  max:  3484.3037
  mean:  134.27835

block3_conv1
  shape:  (1, 106, 128, 256)
  min:  0.0
  max:  7291.078
  mean:  143.77878

block4_conv1
  shape:  (1, 53, 64, 512)
  min:  0.0
  max:  13492.799
  mean:  530.00244

block5_conv1
  shape:  (1, 26, 32, 512)
  min:  0.0
  max:  2881.529
  mean:  40.596397

克矩阵

计算风格

图像的内容由地图共同特征的值表示。

通过在所有位置获取输出乘积来计算包括此信息的克矩阵。

可以针对特定层计算Gram矩阵, 如下所示：

这是使用tf.linalg.einsum函数简洁地实现的：

def gram_matrix(input_tensor):
result = tf.linalg.einsum('bijc, bijd->bcd', input_tensor, input_tensor)
input_shape = tf.shape(input_tensor)
  num_locations = tf.cast(input_shape[1]*input_shape[2], tf.float32)
  return result/(num_locations)

提取图像的样式和内容

构建返回内容和样式张量的模型。

class StyleContentModel(tf.keras.models.Model):
def __init__(self, style_layers, content_layers):
super(StyleContentModel, self).__init__()
self.vgg =  vgg_layers(style_layers + content_layers)
self.style_layers = style_layers
self.content_layers = content_layers
self.num_style_layers = len(style_layers)
self.vgg.trainable = False
def call(self, inputs):
"Expects float input in [0, 1]"
    inputs = inputs*255.0
    preprocessed_input = tf.keras.applications.vgg19.preprocess_input(inputs)
    outputs = self.vgg(preprocessed_input)
    style_outputs, content_outputs = (outputs[:self.num_style_layers], outputs[self.num_style_layers:])
style_outputs = [gram_matrix(style_output)
 for style_output in style_outputs]

    content_dict = {content_name:value for content_name, value in zip(self.content_layers, content_outputs)}
style_dict = {style_name:value
                  for style_name, value
                  in zip(self.style_layers, style_outputs)}
return {'content':content_dict, 'style':style_dict}

当在图像上调用时, 此模型返回style_layers的语法矩阵(样式)和content_layers的内容：

extractor = StyleContentModel(style_layers, content_layers)
results = extractor(tf.constant(content_image))
style_results = results['style']
print('Styles:')
for name, output in sorted(results['style'].items()):
  print("  ", name)
  print("    shape: ", output.numpy().shape)
  print("    min: ", output.numpy().min())
  print("    max: ", output.numpy().max())
  print("    mean: ", output.numpy().mean())
  print()
print("Contents:")
for name, output in sorted(results['content'].items()):
  print("  ", name)
  print("    shape: ", output.numpy().shape)
  print("    min: ", output.numpy().min())
  print("    max: ", output.numpy().max())
  print("    mean: ", output.numpy().mean())

输出

Styles:
   block1_conv1
    shape:  (1, 64, 64)
    min:  0.0055228453
    max:  28014.557
    mean:  263.79025

   block2_conv1
    shape:  (1, 128, 128)
    min:  0.0
    max:  61479.496
    mean:  9100.949

   block3_conv1
    shape:  (1, 256, 256)
    min:  0.0
    max:  545623.44
    mean:  7660.976

   block4_conv1
    shape:  (1, 512, 512)
    min:  0.0
    max:  4320502.0
    mean:  134288.84

   block5_conv1
    shape:  (1, 512, 512)
    min:  0.0
    max:  110005.37
    mean:  1487.0381

Contents:
   block5_conv2
    shape:  (1, 26, 32, 512)
    min:  0.0
    max:  2410.8796
    mean:  13.764149

运行梯度下降

使用此样式和内容提取器, 我们实现了样式转移算法。为此, 请评估图像输出中相对于每个目标的均方误差, 然后对损失进行加权求和。

设置我们的样式和内容目标值：

style_targets = extractor(style_image)['style']
content_targets = extractor(content_image)['content']

定义一个tf.Variable包含要保留的图像。借助内容图像进行初始化(tf.Variable与内容图像具有相同的形状)：

image = tf.Variable(content_image)

这是一个浮动图像, 定义一个函数以将像素值保持在0到1之间：

def clip_0_1(image):
return tf.clip_by_value(image, clip_value_min=0.0, clip_value_max=1.0)

创建优化器。该论文推荐LBFGS：

opt = tf.optimizers.Adam(learning_rate=0.02, beta_1=0.99, epsilon=1e-1)

要对其进行优化, 请使用两个损失的权重组合来获得总损失：

style_weight=1e-2
content_weight=1e4

def style_content_loss(outputs):
    style_outputs = outputs['style']
    content_outputs = outputs['content']
    style_loss = tf.add_n([tf.reduce_mean((style_outputs[name]-style_targets[name])**2) 
                           for name in style_outputs.keys()])
    style_loss *= style_weight / num_style_layers

    content_loss = tf.add_n([tf.reduce_mean((content_outputs[name]-content_targets[name])**2) 
for name in content_outputs.keys()])
    content_loss *= content_weight / num_content_layers
    loss = style_loss + content_loss
    return loss

使用函数tf.GradientTape更新图像。

@tf.function()
def train_step(image):
  with tf.GradientTape() as tape:
 outputs = extractor(image)
 loss = style_content_loss(outputs)
grad = tape.gradient(loss, image)
opt.apply_gradients([(grad, image)])
  image.assign(clip_0_1(image))

运行以下步骤进行测试：

train_step (image)
train_step (image)
train_step (image)
plt.imshow(image.read_value()[0])

输出

变换图像

在此步骤中执行更长的优化：

import time
start = time.time()

epochs = 10
steps_per_epoch = 100
step = 0
for n in range(epochs):
  for m in range(steps_per_epoch):
    step += 1
    train_step(image)
    print(".", end='')
  display.clear_output(wait=True)
  imshow(image.read_value())
  plt.title("Train step: {}".format(step))
plt.show()
end = time.time()
print("Total time: {:.1f}".format(end-start))

输出

总变化损失

def high_pass_x_y(image):
  x_var = image[:, :, 1:, :] - image[:, :, :-1, :]
  y_var = image[:, 1:, :, :] - image[:, :-1, :, :]
   return x_var, y_var

x_deltas, y_deltas = high_pass_x_y(content_image)
plt.figure(figsize=(14, 10))
plt.subplot(2, 2, 1)
imshow(clip_0_1(2*y_deltas+0.5), "Horizontal Deltas: Original")
plt.subplot(2, 2, 2)
imshow(clip_0_1(2*x_deltas+0.5), "Vertical Deltas: Original")
x_deltas, y_deltas = high_pass_x_y(image)
plt.subplot(2, 2, 3)
imshow(clip_0_1(2*y_deltas+0.5), "Horizontal Deltas: Styled")
plt.subplot(2, 2, 4)
imshow(clip_0_1(2*x_deltas+0.5), "Vertical Deltas: Styled")

输出

这表明高频分量是如何增加的。

该高频分量是边缘检测器。在给定的示例中, 我们从边缘检测器获得了相同的输出：

plt.figure(figsize=(14, 10))
sobel = tf.image.sobel_edges(content_image)
plt.subplot(1, 2, 1)
imshow(clip_0_1(sobel [..., 0]/4+0.5), "Horizontal Sobel-edges")
plt.subplot(1, 2, 2)
imshow(clip_0_1(sobel[..., 1]/4+0.5), "Vertical Sobel-edges")

输出

与此相关的正则化损失是值的平方和：

def total_variation_loss(image):
  x_deltas, y_deltas = high_pass_x_y(image)
  return tf.reduce_sum(tf.abs(x_deltas)) + tf.reduce_sum(tf.abs(y_deltas))

total_variation_loss(image).numpy()

输出

99172.59

那证明了它的作用。但是不需要自己实现, 它包括一个标准实现：

tf.image.total_variation(image).numpy()

输出

array([99172.59], dtype=float32)

重新运行优化功能

选择函数total_variation_loss的权重：

total_variation_weight=30

现在, train_step函数：

@tf.function()
def train_step(image):
with tf.GradientTape() as tape:
outputs = extractor(image)
loss = style_content_loss(outputs)
loss += total_variation_weight*tf.image.total_variation(image)
grad = tape.gradient(loss, image)
opt.apply_gradients([(grad, image)])
image.assign(clip_0_1(image))

重新初始化优化变量：

image = tf.Variable(content_image)

并运行优化：

import time
start = time.time()

epochs = 10
steps_per_epoch = 100

step = 0
for n in range(epochs):
  for m in range(steps_per_epoch):
    step += 1
    train_step(image)
print(".", end='')
  display.clear_output(wait=True)
  display.display(tensor_to_image(image))
  print("Train step: {}".format(step))
end = time.time()
print("Total time: {:.1f}".format(end-start))

输出

最终保存结果

file_name = 'styletransfer.png'
tensor_to_image(image). save(file_name)
try: from google. colab import files
except ImportError:
pass
else:
  files.download(file_name)