grammarVAE

file:///C:/Users/PPCC/Downloads/1703.01925.pdf
https://github.com/mkusner/grammarVAE
https://www.iczhiku.com/videoDetail/2FOiHMpJ3@g@r7fpC@bxQtA==

BERT in Keras

想讓電腦快速聽懂人話嗎？

2020的最近釋出的GPT3看起來非常的驚豔。不過由於模型的複雜度讓普通電腦是無法安裝的，相對於GPT3，類似的模型有BERT家族。這邊提供一個快速的體驗方式來使用BERT：

首先先確定電腦有安裝好 keras&tensorflow

• 安裝教學 Ubuntu 18.04+GPU驅動+CUDA10.0+cuDNN7.6.5+Tensorflow-gpu-1.15.0+Keras 2.3.1

https://reurl.cc/Oq7nn7

• Tensorflow、Keras傻瓜式安裝教學

https://reurl.cc/k0Nggr

❷ 安裝社群版的Keras BERT (https://github.com/CyberZHG/keras-bert)

pip install --user keras-bert

❸ 範例程式

假設我們有以下句子

[起] 台 北 是 ？？ 的 首 都 [結束]

要猜的詞是 ？？，如果電腦有能力接近理解，應該猜出是”台灣”或”臺灣”

先把它轉換成list輸入：

[CLS] 台 北 是 [MASK] [MASK] 的 首 都 [SEP]

# Keras BERT API Demo
import sysimport numpy as npfrom keras_bert import (    load_vocabulary,    load_trained_model_from_checkpoint,    Tokenizer,    get_checkpoint_paths,)from keras_bert.datasets import get_pretrained, PretrainedList
# 導入已訓練中文模型model_path = get_pretrained(PretrainedList.chinese_base)paths = get_checkpoint_paths(model_path)model = load_trained_model_from_checkpoint(    paths.config, paths.checkpoint, training=True, seq_len=None)model.summary(line_length=120)# 導入字典token_dict = load_vocabulary(paths.vocab)token_dict_inv = {v: k for k, v in token_dict.items()}# 要測試的字串# 舉例 : CLS 台 北 是 MASK MASK 的 首 都 SEP# MASK = 要猜的字串tokens = ["[CLS]"]mask = ["[MASK]", "[MASK]"]text1 = "台北是"text2 = "的首都"tokens.extend(list(text1))tokens.extend(mask)tokens.extend(list(text2))tokens.append("[SEP]")print("Tokens to predict:", tokens)indices = np.array([[token_dict[token] for token in tokens]])segments = np.array([[0] * len(tokens)])# 要預測的位置為1 其它為0masks = np.array([[0, 0, 0, 0, 1, 1, 0, 0, 0, 0]])predicts = model.predict([indices, segments, masks])[0].argmax(axis=-1).tolist()print("Fill with: ", list(map(lambda x: token_dict_inv[x], predicts[0][4:6])))

❹ 輸出結果

電腦成功預測出[MASK] [MASK] 的位置要放”台灣”



分享自FB社團：台灣自然語言處理與人工智慧交流社（https://www.facebook.com/groups/Taiwan.NLP/）

預訓練模型清單:
https://github.com/CyberZHG/keras-bert/blob/e9ccba8ed56b52102bfd08fd231bf9b79d7e0e3e/keras_bert/datasets/pretrained.py#L15

PretrainedList.wwm_cased_large:
1. Large model,
2. Trained on case-sensitive data,
3. Trained using Whole Word Masking (WWM)

Bert: Step by step by Hugging face

https://medium.com/swlh/bert-step-by-step-b7ff47fcfbe

Keras vs PyTorch，哪一個更適合做深度學習？

https://bangqu.com/ZEfXs9.html?utm_source=Facebook_PicSee&fbclid=IwAR0Ytj6clBd12rISpYvaTJ8ZnFWyXMTJf3GEtD6y7_Qp9ide3spqVO20-to

Keras vs Tensorflow vs Python

https://machinelearningknowledge.ai/keras-vs-tensorflow-vs-pytorch-no-more-confusion/?fbclid=IwAR0V3Xs2UEev7ZKu-yMKVaFOnY-t8TEgvoWVlBQJQAVIMpUqAnAWUmuFLUk

Mac install tensorflow 2.1.0, Keras 2.3.1, scikit-learn, and RDkit

Tensorflow:
我選擇system安裝，不使用virtual environment

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ export PATH="/usr/local/bin:/usr/local/sbin:$PATH"
$ brew update
$ brew install python
$ sudo pip3 install -U virtualenv
$ pip3 install --user --upgrade tensorflow

如果是python3.7.3版本的話要修改linecache.py

$ sudo vi /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/linecache.py

第48行改成
for mod in list(sys.modules.values()): # for mod in sys.modules.values():

Keras:

$ brew install hdf5
$ brew install graphviz
$ sudo pip3 install pydot (一定要加sudo)
$ vi ~/.bash_profile
export PATH=/usr/local/bin:/usr/local/sbin:~/bin:$PATH
$ source ~/.bash_profile
$ sudo pip3 install keras (一定要加sudo)

scikit-learn:
$ pip3 install -U scikit-learn

RDkit:
$ brew tap rdkit/rdkit
$ brew install rdkit

If you need to have icu4c first in your PATH run:
  echo 'export PATH="/usr/local/opt/icu4c/bin:$PATH"' >> ~/.zshrc
  echo 'export PATH="/usr/local/opt/icu4c/sbin:$PATH"' >> ~/.zshrc

For compilers to find icu4c you may need to set:
  export LDFLAGS="-L/usr/local/opt/icu4c/lib"
  export CPPFLAGS="-I/usr/local/opt/icu4c/include"

==> openblas
openblas is keg-only, which means it was not symlinked into /usr/local,
because macOS provides BLAS and LAPACK in the Accelerate framework.

For compilers to find openblas you may need to set:
  export LDFLAGS="-L/usr/local/opt/openblas/lib"
  export CPPFLAGS="-I/usr/local/opt/openblas/include"

==> rdkit
      You may need to add RDBASE to your environment variables.
      For Bash, put something like this in your $HOME/.bashrc:
        export RDBASE=/usr/local/share/RDKit

參考
https://github.com/rdkit/homebrew-rdkit

Face Detection and Recognition with MTCNN model and VGGFace2 in Keras

https://morioh.com/p/a07857cbc76d?fbclid=IwAR029OYbV1XZQEleJowgX4Kt58uTCBgG8kB15GWmItz86ZdYJxOSfpRncpA

Towards a Super Resolution Autoencoder (code included)

https://www.linkedin.com/pulse/towards-super-resolution-autoencoder-code-included-ibrahim-sobh-phd?fbclid=IwAR39_nyGVc0j2y6fbuqhnwPFbUpBntxhY1PoU2UtsFegXlZCxp2NN3Qn1sU

How to Use Transfer Learning for Image Classification using Keras in Python

https://www.thepythoncode.com/article/use-transfer-learning-for-image-flower-classification-keras-python?fbclid=IwAR01Q3AH1G3s_U2dI9DfJFAkkH-38POqgytUIZyQqhKqWaBsKgVUJ2yO1zk

Keras建立custom loss function以及load_model時怎麼呼叫custom loss function

本文介紹訓練神經網路如果是使用客制化的loss function，訓練完後要如何使用load_model呼叫

On the use of the Kullback–Leibler divergence in Variational Autoencoders

可變自動編碼器(VAE)的loss function可以分為兩部分。第一個測量autoencoding的量，即原始樣本與其重構之間的誤差。第二個是有關standard multivariate normal distribution的Kullback-Leibler散度(縮寫為KL散度)。我們將畫圖說明KL散度對encoder 和decoder 輸出的影響。

VAE code

VAE published in ACS Cent. Sci.
https://github.com/aspuru-guzik-group/chemical_vae

Explicit hyperparameter:
https://github.com/maxhodak/keras-molecules

Generating sentences:
http://alexadam.ca/ml/2017/05/05/keras-vae.html?
https://github.com/alexeyev/Keras-Generating-Sentences-from-a-Continuous-Space
https://github.com/psipred/protein-vae
https://github.com/bojone/vae (GCNN)

Protein sequence:
https://github.com/samsinai/VAE_protein_function/blob/master/VAE_for_protein_function_prediction.ipynb
https://github.com/psipred/protein-vae
https://gitlab.com/PhazCode/DeepProteinDesign/tree/master

GAN+RL for Generative-Molecules:
https://github.com/chennnnnyize/Generative-Molecules

keras 中模型的保存及重用

http://frankchen.xyz/2018/04/19/keras-reuse-model/

Step-by-step understanding LSTM Autoencoder layers

https://towardsdatascience.com/step-by-step-understanding-lstm-autoencoder-layers-ffab055b6352

define one-to-one, one-to-many, many-to-one, and many-to-many LSTM neural networks in Keras

https://stackoverflow.com/questions/52138290/how-can-we-define-one-to-one-one-to-many-many-to-one-and-many-to-many-lstm-ne

Python keras.layers.RepeatVector() Examples

https://www.programcreek.com/python/example/89689/keras.layers.RepeatVector

LSTM 類神經網路學習紀錄

http://mark1002.github.io/2018/04/04/LSTM-%E9%A1%9E%E7%A5%9E%E7%B6%93%E7%B6%B2%E8%B7%AF%E5%AD%B8%E7%BF%92%E7%B4%80%E9%8C%84/

LSTM Arguments設定input_shape

為何在Keras Document裡面的Arguments明明沒有input_shape，可是常常會看到有人使用呢?如下所示:

model.add(LSTM(32, input_shape=(10, 64)))

這是因為LSTM是keras.engine.base_layer.wrapped_fn()抽像類別的子類別，所有的循環層(LSTM，GRU，SimpleRNN)都繼承本層，因此下面的參數可以在任何循環層中使用。

• cell: A RNN cell instance. A RNN cell is a class that has:
  • a call(input_at_t, states_at_t) method, returning (output_at_t, states_at_t_plus_1). The call method of the cell can also take the optional argument constants, see section "Note on passing external constants" below.
  • a state_size attribute. This can be a single integer (single state) in which case it is the size of the recurrent state (which should be the same as the size of the cell output). This can also be a list/tuple of integers (one size per state).
  • a output_size attribute. This can be a single integer or a TensorShape, which represent the shape of the output. For backward compatible reason, if this attribute is not available for the cell, the value will be inferred by the first element of the state_size.
  It is also possible for cell to be a list of RNN cell instances, in which cases the cells get stacked one after the other in the RNN, implementing an efficient stacked RNN.
• return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence.
• return_state: Boolean. Whether to return the last state in addition to the output.
• go_backwards: Boolean (default False). If True, process the input sequence backwards and return the reversed sequence.
• stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.
• unroll: Boolean (default False). If True, the network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences.
• input_dim: dimensionality of the input (integer). This argument (or alternatively, the keyword argument input_shape) is required when using this layer as the first layer in a model.
• input_length: Length of input sequences, to be specified when it is constant. This argument is required if you are going to connect Flatten then Dense layers upstream (without it, the shape of the dense outputs cannot be computed). Note that if the recurrent layer is not the first layer in your model, you would need to specify the input length at the level of the first layer (e.g. via the input_shape argument)

參考
https://keras.io/layers/recurrent/
https://keras-cn.readthedocs.io/en/latest/layers/recurrent_layer/

Variational Autoencoder: Intuition and Implementation

目前對於數據生成的神經網路架構存在兩種並駕齊驅的生成模型：生成對抗網絡(GAN)和變異自編碼器(VAE)。 這兩種模型的訓練方式相當不同。 GAN來自於game theory，其目標是找到discriminator網絡與generator 網絡之間的Nash Equilibrium。 另一方面，VAE來自於bayesian inference，也就是建立數據的基本機率分佈模型，以便可以從該分佈中採樣新數據。

在本文中，我們將直觀地研究VAE模型及其在Keras中的實現。

Keras implementation of an LSTM neural network to classify and predict the MINST dataset

這篇將會針對經常用於時間序列數據進行分類的神經網路LSTM對2D數據進行分類，例如手寫字母的圖像來進行討論。