Introduction to Perforce for IFS

Paul Dando
ECMWF User Support
Paul.Dando@ecmwf.int
Slide 1

Introduction to Perforce for IFS Slide 1

Overview

z Source code management

z Perforce commands p4 and q2
z Creating, selecting and copying branches
z Adding, editing and deleting files
z Finding differences between branches
z Merging branches
z Using your branch in prepIFS
z Some simple examples
z Getting help
Slide 2

Introduction to Perforce for IFS Slide 2

1
Source Code Management

z Management of multiple revisions of the same unit of

application source code
- Simplest form: cp myprog.F myprog_v1.F
z Tracks and provides control over changes to source
code
- Versions of files from any moment of time are easily recreated
z Allows multiple developers to work on code concurrently
- File locking or Version merging
z Most use delta compression
- retains only the differences between successive versions of files
- allows more efficient storage of many Slide
different
3 versions of files
z Examples: RCS, CVS, Subversion, ClearCase

Introduction to Perforce for IFS Slide 3

Introduction – what is Perforce ?

z Perforce is the main SCM used at ECMWF

- Replaced ClearCase in 2005
z Perforce is a commercial product
- See http://www.perforce.com/
z Simple client-server SCM system
- Client (p4) talks to a server (p4d) where the master file repository
(or “depot”) is held
z Perforce clients available for all major operating systems
in use at ECMWF
z Use Perforce client commands to communicate with server
- p4 commands are Perforce client commands
Slide 4

- q2 commands are ECMWF local commands

Introduction to Perforce for IFS Slide 4

2
Perforce terminology

z The Perforce server manages the master file repository or

depot
- Contains every revision of every file under Perforce control
- Files in depot referred to as depot files or versioned files
z Each new code line is referred to as a branch
- Each branch is saved in a new unique location in the depot
- An IFS release is also a branch
- For IFS we use a different Perforce client for each branch
z Editing is always done on local copies of the files synced
(extracted) from the depot
- Local files reside in your client workspace
Slide 5
- For Member State users this is $SCRATCH/p4w
- Set by the environment variable $Q2LOCALROOT

Introduction to Perforce for IFS Slide 5

Perforce changelists

z The changelist is the basic unit of work in Perforce

- List of files, revision numbers and changes made
- Each has a Perforce generated changelist number
- Each has user-provided changelist description
z A changelist serves two purposes
- To group related changes together
- To ensure all related changes are checked in together
z Perforce changelists are atomic change transactions
- All files in a changelist are committed or none
z A pending changelist – changes not yet submitted to depot
z A submitted changelist – changes committed
Slide 6
to the depot

Introduction to Perforce for IFS Slide 6

3
Depot structure

project group

//depot/main/ifs/ifs

... /ifs/trans projects Main code line

... /ifs/... Only modified by admin

... /prepifs/prepifs

... /hope/hope

... /...

//depot/rel/ifs/CY31R1 IFS releases

.../ifs/CY31R2 Only modified by admin

.../ifs/... Slide 7 Accessed by users

//depot/user/<user ID>/<branch_name>/ifs User branches

Introduction to Perforce for IFS Slide 7

IFS Projects at Cycle 32r2

IFS Projects IFS Projects Other project

Accessible not accessible groups
to MS users to MS Users accessible to
z aeolus z bl MS Users
z ifs z ssa z hope
z ifsaux z verif
z nemo
z prepdata z obstat
z nemovar
z scripts z odb
z surf z reanal z odasys
z trans z satrad
z wam z scat Slide 8
z scmec

Introduction to Perforce for IFS Slide 8

4
 Branching

//depot/user/<uid>/branch1 //depot/user/<uid>/branch3

//depot/user/<uid>/branch2

Integ
//depot/rel/ifs/CY32R2
//depot/rel/ifs/CY32R1

rate
main code line //depot/main/ifs

z Perforce uses the term “branch” to mean any one of

- a code line
Slide 9
- a branch view specification
- to create a new code line – “to branch”

Introduction to Perforce for IFS Slide 9

Commands p4 and q2

z p4 – Perforce command client

- p4 edit – open file for edit (“check-out”)
- p4 add – informs depot there is a new file
- p4 submit – submit changes to depot (“check-in”)
z q2 – ECMWF perl script on top of p4
- q2 create_branch
- q2 select_client
- q2 merge_branch
Needs /home/rd/rdx/bin added to your $PATH
z To use p4 commands outside the “q2 environment”
- setenv P4PORT p4rd:14000 (C shell)
Slide 10

- export P4PORT=p4rd:14000 (Korn shell)

Introduction to Perforce for IFS Slide 10

5
Getting help

z List all available q2 commands using

q2

z Get help on a specific q2 command with

q2 help <command>
q2 help create_branch

z Get help with p4 commands

p4 help
p4 help commands

p4 help <command> Slide 11

p4 help edit

Introduction to Perforce for IFS Slide 11

Creating a new branch

z Use the q2 create_branch command

Branch name IFS cycle Project
q2 create_branch –b testb_1 –r CY32R2 –p ifs
z Prompts for a branch descriptor
- Text describing the branch
z Provides a list of projects if none specified with “-p”
- Specify more projects by separating with colon, e.g. ifs:script
z Creates a branch with name of form
<uid>_<IFS_CYCLE>_<branch name>
z Downloads files to your local client workspace from
//depot/rel/ifs/<IFS_CYCLE> Slide 12
z Sets your current working directory for this client/branch

Introduction to Perforce for IFS Slide 12

6
Creating a branch – what really happens

depot workspace

//depot/rel/ifs/CY31R1

copy <branch>

nt
//depot/user/<uid>/<branch> clie
y
cop

z Creates a virtual copy in a new location in the depot

z Creates a copy of all the files in your workspace
Slide 13
- Referred to as syncing – gets latest version of files from depot
z Creates a client so that your local copy can “talk” to the depot
Introduction to Perforce for IFS Slide 13

A Perforce client specification

z p4 workspace –o <client_name>
Client: client_us2_CY32R2_new # The client name
Update: 2007/08/30 15:30:27 # Date of last update
Access: 2007/09/18 10:07:42 # Date client last used

Owner: us2 # User that created

Description: # Short description
Client constructed by q2 v1.0
Root: /scratch/ms/gb/us2/p4w/us2_CY32R2_new # Base directory Of
# client workspace
Options: noallwrite noclobber nocompress locked modtime normdir
SubmitOptions: submitunchanged
LineEnd: local
View: # Lines to map depotSlide
files14 into client workspace
//depot/user/us2/us2_CY32R2_new/ifs/... //client_us2_CY32R2_new/ifs/...

Introduction to Perforce for IFS Slide 14

7
Referring to files in Perforce

z Files in Perforce can be specified using

Depot syntax - //depot/user/us2/us2_CY32R2_new/ifs/prog.F90
Client syntax - //client_us2_CY32R2_new/ifs/prog.F90
Local syntax - /scratch/ms/gb/us2/p4w/us2_CY32R2_new/ifs/prog.F90
z When mapping depot files to local filesystem client
workspace is an alias for the client workspace root
//client_us2_CY32R2_new/ Ù /scratch/ms/gb/us2/p4w/us2_CY32R2_new/

client workspace client workspace root

z Depot file
//depot/user/us2/us2_CY32R2_new/ifs/prog.F90
is mapped to local file Slide 15

/scratch/ms/gb/us2/p4w/us2_CY32R2_new/ifs/prog.F90

Introduction to Perforce for IFS Slide 15

Selecting an existing branch

z Use the q2 select_client command

q2 select_client
- Lists all of your branches
- Prompts for branch (“client”) to select
- Syncs files for this branch from depot to local workspace
z Can select another user’s branches with
q2 select_client –u <uid>
- Cannot make changes to another user’s branch

z Dates on files in your local workspace are the dates

when the files were added to the depot
Slide 16

Introduction to Perforce for IFS Slide 16

8
Editing a file

z To open a file for edit (“check-out”) use

p4 edit <file>

z The Perforce client then

- Checks out the file in a changelist
- Makes the file in your workspace writeable
- Informs the Perforce server that the file is opened for editing

z The file can then be edited with your favourite editor

z All edits are local and can be lost until

Slide 17they are

submitted (“checked in”) to the depot with p4 submit

Introduction to Perforce for IFS Slide 17

Adding a new file

z To add a new file

- Create the file in your workspace
ƒ Copy or create using your favourite editor

z Mark the file for adding to the depot in a changelist using

p4 add <file>

z The Perforce client informs the server that you intend to

add a new file to be tracked

z The new file is local only and can be lost

Slide 18 until it is
submitted (“checked in”) to the depot with p4 submit

Introduction to Perforce for IFS Slide 18

9
Deleting a file

z To remove a file use

p4 delete <file>

z The Perforce client

- Marks the file for deletion in a changelist
- Deletes the file immediately from your workspace
- File removed from the depot only on p4 submit

z Deleted file versions are never actually removed from the

depot
- Older revisions of deleted files can be Slide
recovered
19 by syncing
versions that pre-date the deletion to your client workspace

Introduction to Perforce for IFS Slide 19

Discarding unwanted changes

z Changes made to a file in a changelist can be discarded

z To revert changes to open files use
p4 revert <file>

z Reverting a file
- Removes the file from its changelist
- For files marked for edit or delete
ƒ the last submitted version of <file> is copied from the
depot to your client workspace
- For files marked for add
ƒ Slide
the local copy of the file remains in 20
your client workspace
until the next sync with the depot

Introduction to Perforce for IFS Slide 20

10
Submitting your changes
p4 submit
z Submits changes made to the depot
- All changes since last check-in with p4 submit
- Can be one or many files
z Opens the change specification in an editor
- You must enter a change description to replace the line
<enter description here>
z Returns a changelist number
z Increments the version number of all files changed
- call_sl.F90#2 Version number
z Until p4 submit is issued all edits are
Slide local
21 and can be lost !
- Always check in any significant changes !

Introduction to Perforce for IFS Slide 21

A Perforce change specification

Change: new

Client: client_us2_CY32R2_test You must enter text

describing your
User: us2
change here
Status: new

Description: You can remove from

<enter description here> this list any files you
do not want to submit
Files:

//depot/user/us2/us2_CY32R2_new/ifs/nmi/vtran_new.F90 # add

//depot/user/us2/us2_CY32R2_new/ifs/nmi/vtran.F90 # edit

~ Slide 22

Introduction to Perforce for IFS Slide 22

11
Finding out which files have changed

z To list files modified on a branch use

q2 find_files –n

z To list files currently open (by p4 edit)

p4 opened
z To see differences between file versions on the depot
p4 diff2 call_sl.F90#1 call_sl.F90#2
z Or differences with a specific changelist version
p4 diff2 call_sl.F90#2 call_sl.F90@<changelist_number>
z For differences between file versions opened in the
client workspace with those in the depot
p4 diff call_sl.F90 Slide 23

Introduction to Perforce for IFS Slide 23

Getting details of changes made

z To get a revision history list for a particular file

p4 filelog <file>
- Lists versions and changelists for <file>
z To get a list of changelists affecting a file
p4 changes <file>

z To get a list of changelists for a client

p4 changes –c <client_name>

z Can refer to current client using $P4CLIENT

p4 changes –c $P4CLIENT
- Lists changelist numbers and their descriptions
Slide 24

- Indicates whether any of the changelists are pending

Introduction to Perforce for IFS Slide 24

12
Adding projects to an existing branch

z Additional projects can be added to an existing branch

z First select the branch with
q2 select_client –b <branch_name>

z Then add additional projects with e.g.:

Projects IFS Cycle
q2 add_to_client –p ifsaux:surf –r CY32R2

Slide 25

Introduction to Perforce for IFS Slide 25

Copying a branch

z Create a new branch for the required IFS cycle

q2 create_branch –b new_branch –r CY32R2 –p ifs

z Set the visual merge tool to emacs or xdiff

C-shell: setenv P4MERGE p4merge_emacs # or xdiff
Korn shell: export P4MERGE=p4merge_emacs
- Only needed if want to view differences between files

z Merge the branch to be copied with the new branch

q2 merge_branch –b usd_CY32R2_old_branch –r
branch to be copied “resolve”
Slide 26
z If select Merge option the visual merge tool will open…
- May need to set your DISPLAY and enable X access !
Introduction to Perforce for IFS Slide 26

13
Merging branches – resolving conflicts

z When two users edit the same file(s) at the same time,
their changes can conflict
z Perforce requires these conflicts to be resolved
- No changes to files in your changelist are submitted to the depot
until you resolve all conflicts
z The resolve process allows you to decide what should
be done
- Should your file overwrite the other user’s ?
- Should your changes be discarded in favour of the other user’s ?
- Should the two sets of changes be merged into one file ?
z When copying a branch you will need to overwrite
changes in the current branch withSlide
the27 those of the
branch being copied – “accept theirs”
Introduction to Perforce for IFS Slide 27

Reverting to a previous version

z Copy version #<n> of the file from depot to your
workspace with
p4 sync <file>#<n>
z Open the file for edit using
p4 edit <file>
z Use p4 submit to create a pending changelist
p4 submit
- This does not submit but returns a changelist number
z Use q2 merge_branch as before
- When prompted accept your version of the file
- Creates a version #<n+1> of the file which is opened for edit
Slide 28
z Use p4 submit to submit the pending changelist
p4 submit –c <changelist_number>
Introduction to Perforce for IFS Slide 28

14
Un/syncing local workspace and depot

z To bring your client workspace into sync with the depot

use
p4 sync [–f] [<file>]
- If no file is specified all files not opened for edit are copied
- The “-f” option forces the sync – a file on the client workspace
will be overwritten even if it is the correct version
- Useful when files on $SCRATCH have been deleted !

z To remove files from your workspace

q2 unsync_client [-d]
- Only removes files that have been checked in with p4 submit
Slide 29
- The “–d” option deletes the client specification (but not the actual
files on in the depot !)

Introduction to Perforce for IFS Slide 29

Using a new branch for your experiment

z Select prepIFS submenu

- Libraries, Perforce Interface, Debug Switch

z Enter branch name in the field

- BRANCH – Perforce branch for all modifications including scripts

z If need a specific version of branch then add this at

- Q2CHANGELIST
- Changelist number identifying a particular set of modifications
- “All changes up to this set”
z Other prepIFS fields exist for SCRIPTS
Slide 30
BRANCH, OCEAN
BRANCH, etc

Introduction to Perforce for IFS Slide 30

15
Entering your branch in prepIFS

Slide 31

Introduction to Perforce for IFS Slide 31

Compile view in XCdp

z Individual
families per
project

Slide 32

Introduction to Perforce for IFS Slide 32

16
Working with other project groups

z Perforce is also used for managing source code changes

for other project groups
- hope, nemo, nemovar odasys

z In this case, need to specify the project group with the

“–g” option when using e.g., q2 create_branch

z For example, for the hope project group

q2 create_branch -g hope –b myhope –r h2e22 –p hope

Slide 33

Introduction to Perforce for IFS Slide 33

Using the visual client (p4v)

z The visual client (p4v) is a GUI which wraps around p4

commands.
- Available for several UNIX platforms: Linux, Solaris, MacOS, etc.
- Not available on AIX (e.g. ecgate) … but made available via a
remote shell to a dedicated Linux machine.
- You can NOT login to this machine but you do not need to.
z p4v is started using
q2 p4v

z p4v can be used for most tasks in Perforce

z For more information see “Getting started with p4v”
Slide 34
http://www.perforce.com/perforce/doc.072/manuals/p4v-gs/p4v-gs.pdf

Introduction to Perforce for IFS Slide 34

17
p4v basic layout

Workspace Tabs

Slide 35

Introduction to Perforce for IFS Slide 35

Demonstrations

z Create a new branch and change a file

z Make changes to an existing branch

z Copying an existing branch

z Reverting to an old version of a file

Slide 36

Introduction to Perforce for IFS Slide 36

18
Create new branch and change a file

z On ecgate create branch

q2 create_branch –b testb_1 –r CY32R2 –p ifs
Branch name IFS cycle Project

- Sets your current working directory to this client’s workspace

z “cd” to directory containing file to be changed
z Check-out file for edit
p4 edit cpg.F90

z Edit the file with your favourite editor

z Commit the change(s) to the depot
p4 submit
Slide 37
z Exit the current client
exit
Introduction to Perforce for IFS Slide 37

Add a new file to an existing branch

z Use the q2 select_client command to select the branch

q2 select_client –b my_branch1
- Sets your current working directory to this client’s workspace
- Without the “-b” option will list all of your branches
z “cd” to directory where new file is to be added
z Mark the new file for addition to the branch
p4 add newprog.F90
z Edit the file with your favourite editor
z Commit the change(s) to the depot
p4 submit
z Exit the current client Slide 38

exit

Introduction to Perforce for IFS Slide 38

19
Further information

z Introducing Perforce Manual

www.perforce.com/perforce/doc.072/manuals/p4guide/p4guide.pdf
- A very useful quick introduction to Perforce and how it works
z Documentation for Perforce is available at:
www.perforce.com/perforce/technical.html
z See also PrepIFS and Source Code Management (SCM) at:
www.ecmwf.int/services/prepifs/learn/
z ECMWF Quick Perforce guide
www.ecmwf.int/services/prepifs/learn/help/Perforce/p4_quick.pdf
z IFS Documentation: VI Technical and Computational
Procedures CY28r1 – Appendix G: Slide
Perforce
39
User Guide
www.ecmwf.int/research/ifsdocs/CY28r1/pdf_files/Technical_part6.pdf

Introduction to Perforce for IFS Slide 39

20